[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:28 dist-test-slave-ldfr krb5kdc[4059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:28 dist-test-slave-ldfr krb5kdc[4059](info): set up 2 sockets
Jun 20 07:11:28 dist-test-slave-ldfr krb5kdc[4059](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 07:11:29.621121 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:29 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:29.753088 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34571
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:29.866658 4072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:29.867034 4072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:29.867125 4072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:29.871538 4072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:29.871649 4072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:29.871688 4072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:29.871734 4072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:29.871783 4072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:29.878046 4072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:29.879725 4072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:29.880856 4072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:29.890996 4077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:29.891022 4078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:29.891001 4080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:29.891050 4072 server_base.cc:1034] running on GCE node
I20260620 07:11:29.891937 4072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:29.893298 4072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:29.894553 4072 hybrid_clock.cc:648] HybridClock initialized: now 1781939489894515 us; error 55 us; skew 500 ppm
Jun 20 07:11:29 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:29.901356 4072 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:29.902513 4072 webserver.cc:460] Webserver started at http://127.3.245.62:37325/ using document root <none> and password file <none>
I20260620 07:11:29.903151 4072 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:29.903251 4072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:29.903502 4072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:29.905826 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "2cdd5f2cde7c405ea3e371a4167131a3"
format_stamp: "Formatted at 2026-06-20 07:11:29 on dist-test-slave-ldfr"
server_key: "12c9a08123478ec6025619832cdc5589"
server_key_iv: "29d2284d465f9d0242c15fbfffd7cf48"
server_key_version: "encryptionkey@0"
I20260620 07:11:29.906483 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "2cdd5f2cde7c405ea3e371a4167131a3"
format_stamp: "Formatted at 2026-06-20 07:11:29 on dist-test-slave-ldfr"
server_key: "12c9a08123478ec6025619832cdc5589"
server_key_iv: "29d2284d465f9d0242c15fbfffd7cf48"
server_key_version: "encryptionkey@0"
I20260620 07:11:29.910799 4072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:11:29.913614 4087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:29.915084 4072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:29.915238 4072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "2cdd5f2cde7c405ea3e371a4167131a3"
format_stamp: "Formatted at 2026-06-20 07:11:29 on dist-test-slave-ldfr"
server_key: "12c9a08123478ec6025619832cdc5589"
server_key_iv: "29d2284d465f9d0242c15fbfffd7cf48"
server_key_version: "encryptionkey@0"
I20260620 07:11:29.915369 4072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:29.956364 4072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:29.960101 4072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:29.960330 4072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:29.967900 4072 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:34571
I20260620 07:11:29.967917 4139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:34571 every 8 connection(s)
I20260620 07:11:29.968978 4072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:29.970845 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4072
I20260620 07:11:29.971021 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:29.972585 4140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:29.972760 4052 external_mini_cluster.cc:1428] Setting key 38e38aab096da4ec287c33a906f67fa3
I20260620 07:11:29.979475 4140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3: Bootstrap starting.
I20260620 07:11:29.982053 4140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:29.982975 4140 log.cc:824] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:29.985064 4140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3: No bootstrap required, opened a new log
I20260620 07:11:29.989403 4140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } }
I20260620 07:11:29.989682 4140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:29.989748 4140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdd5f2cde7c405ea3e371a4167131a3, State: Initialized, Role: FOLLOWER
I20260620 07:11:29.990242 4140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } }
I20260620 07:11:29.990367 4140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:29.990454 4140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:29.990566 4140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:29.991552 4140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } }
I20260620 07:11:29.991915 4140 leader_election.cc:302] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cdd5f2cde7c405ea3e371a4167131a3; no voters:
I20260620 07:11:29.992237 4140 leader_election.cc:288] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:29.992380 4145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:29.992656 4145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2cdd5f2cde7c405ea3e371a4167131a3, State: Running, Role: LEADER
Jun 20 07:11:29 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:29.992970 4145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } }
I20260620 07:11:29.993706 4140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:29.994930 4147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cdd5f2cde7c405ea3e371a4167131a3. Latest consensus state: current_term: 1 leader_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } } }
I20260620 07:11:29.995098 4147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:29.994930 4146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdd5f2cde7c405ea3e371a4167131a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34571 } } }
I20260620 07:11:29.995195 4146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:29.995517 4154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:29.998917 4154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:30.003924 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:29.976766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.3.245.62:34571)
0620 07:11:29.977331 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:29.977355 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:29.978473 (+ 1118us) server_negotiation.cc:407] Connection header received
0620 07:11:29.982167 (+ 3694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:29.982191 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:29.982643 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:29.983020 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:29.985543 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:29.987064 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:29.988462 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:29.989192 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:29.989356 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:29.994081 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:29.994087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:29.994104 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:29.994142 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:29.999538 (+ 5396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.000391 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.000397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.000403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.000572 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.001059 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.001062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.001064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.001616 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.001838 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.002589 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.002901 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":187,"threads_started":1}
I20260620 07:11:30.006402 4154 catalog_manager.cc:1347] Generated new cluster ID: 9a4e2248bbfc420093ec40558f2ebe72
I20260620 07:11:30.006531 4154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:30.016367 4154 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:30.017798 4154 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:30.025672 4154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cdd5f2cde7c405ea3e371a4167131a3: Generated new TSK 0
I20260620 07:11:30.026713 4154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:30.060738 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:30.191022 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:30.191334 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:30.191383 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:30.196008 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:30.196090 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:30.196156 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:30.202031 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:30.203262 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:30.204407 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:30.215262 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.215250 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.215313 4166 server_base.cc:1034] running on GCE node
W20260620 07:11:30.215255 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.215996 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:30.216785 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:30.217990 4166 hybrid_clock.cc:648] HybridClock initialized: now 1781939490217947 us; error 57 us; skew 500 ppm
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:30.221326 4166 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:30.222448 4166 webserver.cc:460] Webserver started at http://127.3.245.1:35339/ using document root <none> and password file <none>
I20260620 07:11:30.223044 4166 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:30.223119 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:30.223306 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:30.225700 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "38f3dd8827f5410e98d54a023393f6d0"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "e20d4c202107189c9112d1a3c4405f46"
server_key_iv: "bf82db8777b26c872037deb33f998685"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.226298 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "38f3dd8827f5410e98d54a023393f6d0"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "e20d4c202107189c9112d1a3c4405f46"
server_key_iv: "bf82db8777b26c872037deb33f998685"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.230854 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 07:11:30.233836 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.235357 4166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:30.235486 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "38f3dd8827f5410e98d54a023393f6d0"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "e20d4c202107189c9112d1a3c4405f46"
server_key_iv: "bf82db8777b26c872037deb33f998685"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.235599 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:30.255488 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:30.259141 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:30.259322 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:30.259903 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:30.260761 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:30.260812 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.260854 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:30.260877 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.271299 4166 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:41167
I20260620 07:11:30.271409 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:41167 every 8 connection(s)
I20260620 07:11:30.272418 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:30.278956 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4166
I20260620 07:11:30.279084 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:30.279336 4052 external_mini_cluster.cc:1428] Setting key c827660a0b2d32b6bb38fb89ee6a756c
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:11:30.290283 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.274486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:59789 (local address 127.3.245.62:34571)
0620 07:11:30.274668 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.274673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.275605 (+ 932us) server_negotiation.cc:407] Connection header received
0620 07:11:30.277113 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.277116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.277168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.277259 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.278748 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.279579 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.281398 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.281875 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.281993 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.285122 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.285129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.285136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.285175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.287330 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.288043 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.288049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.288053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.288159 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.288586 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.288592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.288595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.288906 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.289025 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.289879 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.290037 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:11:30.291175 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.274909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34571 (local address 127.3.245.1:59789)
0620 07:11:30.275433 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:30.275479 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:30.276845 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:30.277570 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:30.277577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:30.277985 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:30.278514 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.278528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.279748 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.279751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.281268 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.281278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.282147 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.282150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.282321 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.283074 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:30.283093 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:30.284899 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:30.287529 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.287535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.287552 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.287893 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.288288 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.288290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.288292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.288459 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.289056 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:30.289062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:30.289379 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 07:11:30.289785 (+ 406us) client_negotiation.cc:240] Negotiation successful
0620 07:11:30.290110 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":117,"threads_started":1}
I20260620 07:11:30.292716 4295 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34571
I20260620 07:11:30.292977 4295 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:11:30.293793 4295 heartbeater.cc:511] Master 127.3.245.62:34571 requested a full tablet report, sending...
I20260620 07:11:30.295733 4103 ts_manager.cc:194] Registered new tserver with Master: 38f3dd8827f5410e98d54a023393f6d0 (127.3.245.1:41167)
I20260620 07:11:30.297619 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:59789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:30.312417 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:30.435732 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:30.436312 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:30.436453 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:30.440955 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:30.441026 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:30.441126 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:30.447064 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:30.448300 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:30.449486 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:30.461311 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.461239 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.461239 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.461382 4300 server_base.cc:1034] running on GCE node
I20260620 07:11:30.462037 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:30.462817 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:30.464043 4300 hybrid_clock.cc:648] HybridClock initialized: now 1781939490463930 us; error 139 us; skew 500 ppm
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:30.467294 4300 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:30.468217 4300 webserver.cc:460] Webserver started at http://127.3.245.2:40215/ using document root <none> and password file <none>
I20260620 07:11:30.468873 4300 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:30.468945 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:30.469118 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:30.471247 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "244cd9dd5993471a9c5e13c0c7951559"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "8423597ac4de707c801eb1828b8fe618"
server_key_iv: "10ee842b77c952103b123090761667e0"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.471796 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "244cd9dd5993471a9c5e13c0c7951559"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "8423597ac4de707c801eb1828b8fe618"
server_key_iv: "10ee842b77c952103b123090761667e0"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.475950 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:11:30.479085 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.480873 4300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:30.480995 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "244cd9dd5993471a9c5e13c0c7951559"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "8423597ac4de707c801eb1828b8fe618"
server_key_iv: "10ee842b77c952103b123090761667e0"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.481154 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:30.491107 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:30.494683 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:30.494920 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:30.495594 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:30.496572 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:30.496675 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.496770 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:30.496798 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.507232 4300 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:32913
I20260620 07:11:30.507251 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:32913 every 8 connection(s)
I20260620 07:11:30.508229 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:30.509548 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4300
I20260620 07:11:30.509663 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:30.509938 4052 external_mini_cluster.cc:1428] Setting key ae097350eef45a56aa349ba8a1a5cc32
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:11:30.526122 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.510494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:44989 (local address 127.3.245.62:34571)
0620 07:11:30.510662 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.510668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.511698 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 07:11:30.513319 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.513324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.513388 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.513484 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.514787 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.515442 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.517459 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.517733 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.517853 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.521783 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.521787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.521789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.521818 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.523432 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.524075 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.524079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.524081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.524178 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.524579 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.524582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.524584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.524916 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.525041 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.525701 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.525867 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:11:30.527192 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.511003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34571 (local address 127.3.245.2:44989)
0620 07:11:30.511522 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:30.511581 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:30.513088 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:30.513729 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:30.513738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:30.514135 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:30.514634 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.514648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.515611 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.515618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.517289 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.517304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.517931 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.517938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.518231 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.519486 (+ 1255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:30.519516 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:30.521514 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:30.523587 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.523593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.523610 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.523921 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.524278 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.524283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.524284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.524461 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.525054 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:30.525064 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:30.525362 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 07:11:30.525599 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 07:11:30.525983 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":158,"threads_started":1}
I20260620 07:11:30.528496 4429 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34571
I20260620 07:11:30.528813 4429 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:30.529520 4429 heartbeater.cc:511] Master 127.3.245.62:34571 requested a full tablet report, sending...
I20260620 07:11:30.530905 4103 ts_manager.cc:194] Registered new tserver with Master: 244cd9dd5993471a9c5e13c0c7951559 (127.3.245.2:32913)
I20260620 07:11:30.531546 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:44989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:30.545625 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:30.671768 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:30.672076 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:30.672122 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:30.676440 4434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:30.676513 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:30.676604 4434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:30.682101 4434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34571
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:30.683357 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:30.684859 4434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:30.695294 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:30.695291 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.695348 4434 server_base.cc:1034] running on GCE node
W20260620 07:11:30.695302 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:30.695935 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:30.696750 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:30.697950 4434 hybrid_clock.cc:648] HybridClock initialized: now 1781939490697917 us; error 51 us; skew 500 ppm
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:30.701314 4434 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:30.702296 4434 webserver.cc:460] Webserver started at http://127.3.245.3:33815/ using document root <none> and password file <none>
I20260620 07:11:30.702862 4434 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:30.702922 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:30.703157 4434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:30.705374 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "a90ea8b342f145538289fe24f5c47afb"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "dd59c85ab761a0ecea6935303fb816d8"
server_key_iv: "6dc6ac4c1ef553ba4284a7f83e92edc3"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.705960 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "a90ea8b342f145538289fe24f5c47afb"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "dd59c85ab761a0ecea6935303fb816d8"
server_key_iv: "6dc6ac4c1ef553ba4284a7f83e92edc3"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.709919 4434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 07:11:30.712862 4449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.714346 4434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:30.714464 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "a90ea8b342f145538289fe24f5c47afb"
format_stamp: "Formatted at 2026-06-20 07:11:30 on dist-test-slave-ldfr"
server_key: "dd59c85ab761a0ecea6935303fb816d8"
server_key_iv: "6dc6ac4c1ef553ba4284a7f83e92edc3"
server_key_version: "encryptionkey@0"
I20260620 07:11:30.714568 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:30.730441 4434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:30.734045 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:30.734206 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:30.734771 4434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:30.735601 4434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:30.735647 4434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.735687 4434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:30.735702 4434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:30.745913 4434 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:36893
I20260620 07:11:30.745944 4562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:36893 every 8 connection(s)
I20260620 07:11:30.746870 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:30.754000 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4434
I20260620 07:11:30.754148 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:30.754413 4052 external_mini_cluster.cc:1428] Setting key f773e2709d4b8ac6c0431f1a15923cf2
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:30.762734 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.748891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:42987 (local address 127.3.245.62:34571)
0620 07:11:30.749031 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.749036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.749972 (+ 936us) server_negotiation.cc:407] Connection header received
0620 07:11:30.751417 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.751420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.751481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.751563 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.752780 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.753326 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.754926 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.755243 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.755372 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.758311 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.758315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.758326 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.758352 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.760058 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.760655 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.760659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.760660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.760768 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.761437 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.761441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.761442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.761701 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.761806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.762433 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.762568 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:11:30.763603 4565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.749330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34571 (local address 127.3.245.3:42987)
0620 07:11:30.749811 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:30.749854 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:30.751193 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:30.751735 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:30.751742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:30.752109 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:30.752599 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.752645 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.753456 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.753460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.754740 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:30.754752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.755411 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.755414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:30.755651 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.756383 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:30.756410 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:30.758132 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:30.760214 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.760220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.760234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.760508 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.760931 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:30.760935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:30.760936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:30.761300 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:30.761842 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:30.761857 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:30.762164 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 07:11:30.762346 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 07:11:30.762635 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":131,"threads_started":1}
I20260620 07:11:30.765053 4563 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34571
I20260620 07:11:30.765332 4563 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:30.765864 4563 heartbeater.cc:511] Master 127.3.245.62:34571 requested a full tablet report, sending...
I20260620 07:11:30.767052 4103 ts_manager.cc:194] Registered new tserver with Master: a90ea8b342f145538289fe24f5c47afb (127.3.245.3:36893)
I20260620 07:11:30.767854 4103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:42987
I20260620 07:11:30.769574 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:11:30.799171 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.787124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.3.245.1:41167)
0620 07:11:30.787466 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.787483 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.787623 (+ 140us) server_negotiation.cc:407] Connection header received
0620 07:11:30.787818 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.787824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.788072 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.788236 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.789038 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.789826 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.790872 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.791313 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.791464 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.794695 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.794711 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.794737 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.794843 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.797067 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.797580 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.797585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.797587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.797678 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.798011 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.798014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.798015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.798283 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.798457 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.798782 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.798999 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":108,"threads_started":1}
W20260620 07:11:30.800278 4209 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57350
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:11:30.829329 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:30.820080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.3.245.1:41167)
0620 07:11:30.820249 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:30.820253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:30.820355 (+ 102us) server_negotiation.cc:407] Connection header received
0620 07:11:30.820581 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:30.820584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:30.820680 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:30.820763 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:30.821565 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.822111 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.822977 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:30.823216 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:30.823330 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:30.826056 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:30.826059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:30.826062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:30.826087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:30.827622 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.828117 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.828120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.828121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.828198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:30.828435 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:30.828437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:30.828438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:30.828712 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:30.828825 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:30.829100 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:11:30.829201 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:11:30.830421 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4166
I20260620 07:11:30.836493 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4300
I20260620 07:11:30.844575 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4434
I20260620 07:11:30.850633 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4072
2026-06-20T07:11:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2305 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4577](info): set up 2 sockets
Jun 20 07:11:30 dist-test-slave-ldfr krb5kdc[4577](info): commencing operation
krb5kdc: starting...
W20260620 07:11:31.924899 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:32.066931 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35623
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:32.191979 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:32.192281 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:32.192323 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:32.196339 4590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:32.196403 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:32.196424 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:32.196446 4590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:32.196462 4590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:32.201959 4590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:32.203055 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:32.204078 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:11:32.213189 4590 server_base.cc:1034] running on GCE node
W20260620 07:11:32.213157 4598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.213210 4596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.213408 4595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.213989 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:32.215145 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:32.216339 4590 hybrid_clock.cc:648] HybridClock initialized: now 1781939492216325 us; error 33 us; skew 500 ppm
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:32.219336 4590 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:32.220263 4590 webserver.cc:460] Webserver started at http://127.3.245.62:41369/ using document root <none> and password file <none>
I20260620 07:11:32.220875 4590 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:32.220940 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:32.221109 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:32.223127 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "44f54d89cc884ff6a76b1b4f811cf89e"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "f3236a41863d2a4e1db865efc5da0e7b"
server_key_iv: "34e5206f101cdded6e8a1ac78939f320"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.223655 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "44f54d89cc884ff6a76b1b4f811cf89e"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "f3236a41863d2a4e1db865efc5da0e7b"
server_key_iv: "34e5206f101cdded6e8a1ac78939f320"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.227523 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:32.230190 4605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.231710 4590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:32.231854 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "44f54d89cc884ff6a76b1b4f811cf89e"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "f3236a41863d2a4e1db865efc5da0e7b"
server_key_iv: "34e5206f101cdded6e8a1ac78939f320"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.231986 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:32.249125 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:32.252810 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:32.253041 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:32.261301 4590 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:35623
I20260620 07:11:32.261318 4657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:35623 every 8 connection(s)
I20260620 07:11:32.262462 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:32.265522 4658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:32.265542 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4590
I20260620 07:11:32.265686 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:32.266013 4052 external_mini_cluster.cc:1428] Setting key d909406bac17006437924fc5eff02451
I20260620 07:11:32.272907 4658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e: Bootstrap starting.
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:32.275722 4658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:32.276562 4658 log.cc:824] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:32.278797 4658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e: No bootstrap required, opened a new log
I20260620 07:11:32.281569 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:32.267345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.3.245.62:35623)
0620 07:11:32.267780 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:32.267796 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:32.267836 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:11:32.268573 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:32.268592 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:32.269036 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:32.269431 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:32.270279 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.271454 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.272399 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.272996 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.273272 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:32.276125 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:32.276131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:32.276147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:32.276179 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:32.278053 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.278557 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.278561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.278566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.278682 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.279017 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.279020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.279022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.279579 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:32.279767 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:32.280133 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 07:11:32.280450 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":152,"threads_started":1}
I20260620 07:11:32.284336 4658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } }
I20260620 07:11:32.284571 4658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:32.284655 4658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f54d89cc884ff6a76b1b4f811cf89e, State: Initialized, Role: FOLLOWER
I20260620 07:11:32.285068 4658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } }
I20260620 07:11:32.285185 4658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:32.285231 4658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:32.285302 4658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:32.286228 4658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } }
I20260620 07:11:32.286533 4658 leader_election.cc:302] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f54d89cc884ff6a76b1b4f811cf89e; no voters:
I20260620 07:11:32.286811 4658 leader_election.cc:288] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:32.287092 4663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:32.287308 4663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [term 1 LEADER]: Becoming Leader. State: Replica: 44f54d89cc884ff6a76b1b4f811cf89e, State: Running, Role: LEADER
I20260620 07:11:32.287622 4663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } }
I20260620 07:11:32.288125 4658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:32.289480 4664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } } }
I20260620 07:11:32.289649 4664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:32.289796 4665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f54d89cc884ff6a76b1b4f811cf89e. Latest consensus state: current_term: 1 leader_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f54d89cc884ff6a76b1b4f811cf89e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35623 } } }
I20260620 07:11:32.289948 4665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:32.290061 4673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:32.293277 4673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:32.298570 4673 catalog_manager.cc:1347] Generated new cluster ID: f2a29473510648c394e37027b09ee579
I20260620 07:11:32.298645 4673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:32.308877 4673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:32.310204 4673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:32.324406 4673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f54d89cc884ff6a76b1b4f811cf89e: Generated new TSK 0
I20260620 07:11:32.325343 4673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:32.360517 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:32.481702 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:32.482028 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:32.482077 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:32.486565 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:32.486644 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:32.486754 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:32.492862 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:32.494052 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:32.495123 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:32.505261 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.505412 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.505265 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.505738 4684 server_base.cc:1034] running on GCE node
I20260620 07:11:32.506227 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:32.506935 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:32.508132 4684 hybrid_clock.cc:648] HybridClock initialized: now 1781939492508110 us; error 30 us; skew 500 ppm
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:32.511294 4684 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:32.512328 4684 webserver.cc:460] Webserver started at http://127.3.245.1:36977/ using document root <none> and password file <none>
I20260620 07:11:32.513015 4684 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:32.513079 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:32.513306 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:32.515484 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "84d25d42fd184fd0a93d20554c1e3c20"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "403d8d2e0dd0082d0af78756a696e29d"
server_key_iv: "80a94e8e9fcc6be5bbb99f5e278bcd77"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.516075 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "84d25d42fd184fd0a93d20554c1e3c20"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "403d8d2e0dd0082d0af78756a696e29d"
server_key_iv: "80a94e8e9fcc6be5bbb99f5e278bcd77"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.520246 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 07:11:32.523468 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.524953 4684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:11:32.525141 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "84d25d42fd184fd0a93d20554c1e3c20"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "403d8d2e0dd0082d0af78756a696e29d"
server_key_iv: "80a94e8e9fcc6be5bbb99f5e278bcd77"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.525280 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:32.551256 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:32.554718 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:32.554965 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:32.555660 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:32.556542 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:32.556613 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.556733 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:32.556783 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.566778 4684 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:36053
I20260620 07:11:32.566797 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:36053 every 8 connection(s)
I20260620 07:11:32.567729 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:32.568015 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4684
I20260620 07:11:32.568158 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:32.568465 4052 external_mini_cluster.cc:1428] Setting key 6a17a70427fa220720ddad7c8cbcc8b7
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:32.587402 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:32.569757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:53829 (local address 127.3.245.62:35623)
0620 07:11:32.569916 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:32.569923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:32.570957 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 07:11:32.572976 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:32.572979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:32.573028 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:32.573108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:32.574906 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.575483 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.577593 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.577857 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.578017 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:32.583026 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:32.583029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:32.583031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:32.583061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:32.584939 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.585498 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.585505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.585509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.585611 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.585958 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.585965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.585969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.586268 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:32.586380 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:32.586973 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 07:11:32.587147 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:11:32.588184 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:32.570299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35623 (local address 127.3.245.1:53829)
0620 07:11:32.570796 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:32.570850 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:32.572705 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:32.573341 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:32.573354 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:32.573920 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:32.574703 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:32.574729 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.575655 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.575661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:32.577283 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:32.577297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.578113 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.578118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:32.578372 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:32.579181 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:32.579205 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:32.582797 (+ 3592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:32.585074 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:32.585080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:32.585094 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:32.585369 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:32.585702 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:32.585705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:32.585706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:32.585841 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:32.586384 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:32.586389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:32.586679 (+ 290us) client_negotiation.cc:769] Sending connection context
0620 07:11:32.586845 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 07:11:32.587137 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260620 07:11:32.589728 4813 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35623
I20260620 07:11:32.590049 4813 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:32.590741 4813 heartbeater.cc:511] Master 127.3.245.62:35623 requested a full tablet report, sending...
I20260620 07:11:32.592586 4622 ts_manager.cc:194] Registered new tserver with Master: 84d25d42fd184fd0a93d20554c1e3c20 (127.3.245.1:36053)
I20260620 07:11:32.593999 4622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:53829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:32.601166 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:32.713359 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:32.713625 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:32.713670 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:32.717577 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:32.717630 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:32.717690 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:32.722879 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:32.723920 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:32.724942 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:32.735169 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.735162 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.735155 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.735548 4818 server_base.cc:1034] running on GCE node
I20260620 07:11:32.735957 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:32.736840 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:32.738054 4818 hybrid_clock.cc:648] HybridClock initialized: now 1781939492738035 us; error 35 us; skew 500 ppm
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:32.741132 4818 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:32.742069 4818 webserver.cc:460] Webserver started at http://127.3.245.2:37123/ using document root <none> and password file <none>
I20260620 07:11:32.742654 4818 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:32.742718 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:32.742895 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:32.745234 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "1676e77a4f14469992b54f1979117546"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "2a00ef81a17d4087e3f285a9a8133d6d"
server_key_iv: "4a8760852fc5cff2464d332374cd1e28"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.745814 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "1676e77a4f14469992b54f1979117546"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "2a00ef81a17d4087e3f285a9a8133d6d"
server_key_iv: "4a8760852fc5cff2464d332374cd1e28"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.750161 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:32.752982 4833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.754388 4818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:32.754571 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "1676e77a4f14469992b54f1979117546"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "2a00ef81a17d4087e3f285a9a8133d6d"
server_key_iv: "4a8760852fc5cff2464d332374cd1e28"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.754695 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:32.776221 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:32.779754 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:32.779978 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:32.780669 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:32.781549 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:32.781605 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.781643 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:32.781699 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:32.791728 4818 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44445
I20260620 07:11:32.791746 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44445 every 8 connection(s)
I20260620 07:11:32.792830 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:32.798162 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4818
I20260620 07:11:32.798303 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:32.798573 4052 external_mini_cluster.cc:1428] Setting key 002ac5ab8b576aadc9d8af8382391747
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:11:32.808275 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:32.794932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:54483 (local address 127.3.245.62:35623)
0620 07:11:32.795097 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:32.795102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:32.795872 (+ 770us) server_negotiation.cc:407] Connection header received
0620 07:11:32.797416 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:32.797419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:32.797483 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:32.797602 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:32.799014 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.799644 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.801042 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.801457 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.801580 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:32.804465 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:32.804468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:32.804470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:32.804496 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:32.806061 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.806648 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.806652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.806654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.806742 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:32.807116 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:32.807119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:32.807121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:32.807313 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:32.807419 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:32.807969 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 07:11:32.808107 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:32.809260 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:32.795188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35623 (local address 127.3.245.2:54483)
0620 07:11:32.795691 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:32.795751 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:32.797182 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:32.797828 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:32.797839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:32.798253 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:32.798863 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:32.798879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.799829 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.799832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:32.800901 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:32.800911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:32.801656 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:32.801662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:32.801898 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:32.802685 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:32.802711 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:32.804267 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:32.806195 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:32.806202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:32.806221 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:32.806536 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:32.806844 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:32.806847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:32.806850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:32.807001 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:32.807429 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:32.807437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:32.807718 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 07:11:32.807913 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 07:11:32.808287 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":134,"threads_started":1}
I20260620 07:11:32.810678 4947 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35623
I20260620 07:11:32.810914 4947 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:11:32.811491 4947 heartbeater.cc:511] Master 127.3.245.62:35623 requested a full tablet report, sending...
I20260620 07:11:32.812736 4622 ts_manager.cc:194] Registered new tserver with Master: 1676e77a4f14469992b54f1979117546 (127.3.245.2:44445)
I20260620 07:11:32.813560 4622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:54483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:32.828471 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:32.961843 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:32.962132 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:32.962188 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:32.966288 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:32.966363 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:32.966429 4952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:32.971873 4952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35623
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:32.973145 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:32.974336 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:32.987699 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.987694 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:32.987715 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:32.987869 4952 server_base.cc:1034] running on GCE node
I20260620 07:11:32.988579 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:32.989485 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:32.990680 4952 hybrid_clock.cc:648] HybridClock initialized: now 1781939492990656 us; error 35 us; skew 500 ppm
Jun 20 07:11:32 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:32.993647 4952 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:32.994555 4952 webserver.cc:460] Webserver started at http://127.3.245.3:45871/ using document root <none> and password file <none>
I20260620 07:11:32.995174 4952 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:32.995260 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:32.995482 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:32.997597 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "bea8eb3f29f8426daecebe6333ebd524"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "be279acb9b163f20f363e0fb3dde6ed2"
server_key_iv: "2563009af70c056f3e069677df562ddb"
server_key_version: "encryptionkey@0"
I20260620 07:11:32.998178 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "bea8eb3f29f8426daecebe6333ebd524"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "be279acb9b163f20f363e0fb3dde6ed2"
server_key_iv: "2563009af70c056f3e069677df562ddb"
server_key_version: "encryptionkey@0"
I20260620 07:11:33.002432 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:33.004966 4967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.006454 4952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:33.006592 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "bea8eb3f29f8426daecebe6333ebd524"
format_stamp: "Formatted at 2026-06-20 07:11:32 on dist-test-slave-ldfr"
server_key: "be279acb9b163f20f363e0fb3dde6ed2"
server_key_iv: "2563009af70c056f3e069677df562ddb"
server_key_version: "encryptionkey@0"
I20260620 07:11:33.006699 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:33.029963 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:33.033339 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:33.033552 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:33.034150 4952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:33.035055 4952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:33.035104 4952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.035144 4952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:33.035192 4952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:33.044176 4952 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:44783
I20260620 07:11:33.044231 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:44783 every 8 connection(s)
I20260620 07:11:33.045187 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:33.045857 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 4952
I20260620 07:11:33.045989 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:33.046276 4052 external_mini_cluster.cc:1428] Setting key 940db0e1b13c150ad949cad117f444f8
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:33.060956 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.047203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:36589 (local address 127.3.245.62:35623)
0620 07:11:33.047363 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.047368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.048394 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 07:11:33.049875 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.049878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.049929 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.050014 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.051283 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.051839 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.053197 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.053478 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.053581 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.056258 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.056261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.056263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.056290 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.058103 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.058711 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.058715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.058716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.058813 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.059409 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.059414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.059416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.059684 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.059770 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.060559 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.060729 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:11:33.061837 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.047681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35623 (local address 127.3.245.3:36589)
0620 07:11:33.048197 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.048249 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.049633 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.050227 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.050234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.050602 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:33.051099 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.051114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.052021 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.052024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.053036 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.053047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.053643 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.053646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.053794 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.054454 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:33.054471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:33.056079 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:33.058245 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.058251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.058266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.058583 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.058989 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:33.058995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:33.058997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:33.059274 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:33.059797 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:33.059807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:33.060254 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.060467 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.060843 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260620 07:11:33.063089 5081 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35623
I20260620 07:11:33.063364 5081 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:33.063855 5081 heartbeater.cc:511] Master 127.3.245.62:35623 requested a full tablet report, sending...
I20260620 07:11:33.065276 4622 ts_manager.cc:194] Registered new tserver with Master: bea8eb3f29f8426daecebe6333ebd524 (127.3.245.3:44783)
I20260620 07:11:33.065855 4622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:36589
I20260620 07:11:33.073521 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:33.085496 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.077451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.3.245.62:35623)
0620 07:11:33.077586 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.077590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.077717 (+ 127us) server_negotiation.cc:407] Connection header received
0620 07:11:33.077843 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.077845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.077887 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.077977 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.078778 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.079419 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.080256 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.080511 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.080699 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.082149 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.082152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.082155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.082182 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.083638 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.084142 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.084145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.084146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.084231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.084542 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.084544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.084545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.084859 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.084980 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.085257 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.085366 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:11:33.091486 4622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:33.094760 4622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:33.110690 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.104855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:36053 (local address 127.0.0.1:44824)
0620 07:11:33.105402 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.105455 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.105593 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.106078 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.106084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.106106 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.106237 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.106244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.107738 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.107741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.109074 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.109084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.110021 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.110024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.110142 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.110185 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.110311 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.110507 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":23,"spinlock_wait_cycles":169984,"thread_start_us":94,"threads_started":1}
I20260620 07:11:33.110740 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.105027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.3.245.1:36053)
0620 07:11:33.105303 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.105310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.105611 (+ 301us) server_negotiation.cc:407] Connection header received
0620 07:11:33.105733 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.105738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.105923 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.106076 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.106385 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.107605 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.109198 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.109885 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.109999 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.110151 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.110412 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.110584 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260620 07:11:33.110817 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.104200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44445 (local address 127.0.0.1:42138)
0620 07:11:33.105402 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.105456 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.105592 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.106631 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.106634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.106652 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.106747 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.106753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.108072 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.108075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.109384 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.109395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.110414 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.110417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.110511 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.110532 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.110591 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.110642 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":23,"spinlock_wait_cycles":611712,"thread_start_us":103,"threads_started":1}
I20260620 07:11:33.111770 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.104405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.3.245.2:44445)
0620 07:11:33.105514 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.105530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.105555 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:11:33.106263 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.106270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.106458 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.106632 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.106879 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.107955 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.109529 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.110285 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.110905 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.111465 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.111518 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.111617 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":139,"threads_started":1}
I20260620 07:11:33.112237 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.104156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:44783 (local address 127.0.0.1:52972)
0620 07:11:33.105754 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.105772 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.105880 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.106285 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.106289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.106306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.106428 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.106438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.108387 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.108391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.109536 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.109545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.110661 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.110665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.110775 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.110799 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.112033 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.112115 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":80,"threads_started":1}
I20260620 07:11:33.113027 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.104244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.3.245.3:44783)
0620 07:11:33.105682 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.105697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.105850 (+ 153us) server_negotiation.cc:407] Connection header received
0620 07:11:33.105990 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.105995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.106159 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.106307 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.106542 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.108232 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.109660 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.110474 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.112539 (+ 2065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.112719 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.112762 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.112836 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":101,"threads_started":1}
I20260620 07:11:33.114395 4881 tablet_service.cc:1467] Processing CreateTablet for tablet b19df0b515b048418a0f4df3d3124d08 (DEFAULT_TABLE table=test-table [id=2f290588ee6243fca6954a643c061e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.114394 4747 tablet_service.cc:1467] Processing CreateTablet for tablet b19df0b515b048418a0f4df3d3124d08 (DEFAULT_TABLE table=test-table [id=2f290588ee6243fca6954a643c061e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.115556 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df0b515b048418a0f4df3d3124d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.115556 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df0b515b048418a0f4df3d3124d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.115367 5015 tablet_service.cc:1467] Processing CreateTablet for tablet b19df0b515b048418a0f4df3d3124d08 (DEFAULT_TABLE table=test-table [id=2f290588ee6243fca6954a643c061e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:33.116360 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19df0b515b048418a0f4df3d3124d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:33.122406 5098 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Bootstrap starting.
I20260620 07:11:33.123551 5099 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Bootstrap starting.
I20260620 07:11:33.124960 5100 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Bootstrap starting.
I20260620 07:11:33.125012 5098 tablet_bootstrap.cc:654] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.125563 5099 tablet_bootstrap.cc:654] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.126332 5098 log.cc:824] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.126457 5099 log.cc:824] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.127660 5100 tablet_bootstrap.cc:654] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:33.128448 5100 log.cc:824] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:33.128851 5098 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: No bootstrap required, opened a new log
I20260620 07:11:33.129063 5098 ts_tablet_manager.cc:1397] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:11:33.129381 5099 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: No bootstrap required, opened a new log
I20260620 07:11:33.129614 5099 ts_tablet_manager.cc:1397] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:33.130779 5100 tablet_bootstrap.cc:492] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: No bootstrap required, opened a new log
I20260620 07:11:33.130982 5100 ts_tablet_manager.cc:1397] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:33.133706 5098 raft_consensus.cc:348] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.134179 5098 raft_consensus.cc:374] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.134282 5098 raft_consensus.cc:729] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea8eb3f29f8426daecebe6333ebd524, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.134387 5099 raft_consensus.cc:348] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.134687 5099 raft_consensus.cc:374] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.134755 5099 raft_consensus.cc:729] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d25d42fd184fd0a93d20554c1e3c20, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.134824 5098 consensus_queue.cc:260] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.135167 5100 raft_consensus.cc:348] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.135288 5099 consensus_queue.cc:260] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.135553 5100 raft_consensus.cc:374] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:33.135668 5081 heartbeater.cc:503] Master 127.3.245.62:35623 was elected leader, sending a full tablet report...
I20260620 07:11:33.135686 5100 raft_consensus.cc:729] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1676e77a4f14469992b54f1979117546, State: Initialized, Role: FOLLOWER
I20260620 07:11:33.136091 5098 ts_tablet_manager.cc:1428] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:11:33.136322 5099 ts_tablet_manager.cc:1428] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 07:11:33.136322 5100 consensus_queue.cc:260] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.136801 4813 heartbeater.cc:503] Master 127.3.245.62:35623 was elected leader, sending a full tablet report...
I20260620 07:11:33.137328 4947 heartbeater.cc:503] Master 127.3.245.62:35623 was elected leader, sending a full tablet report...
I20260620 07:11:33.137571 5100 ts_tablet_manager.cc:1428] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:11:33.232865 5104 raft_consensus.cc:482] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:33.233114 5104 raft_consensus.cc:504] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.234294 5104 leader_election.cc:288] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1676e77a4f14469992b54f1979117546 (127.3.245.2:44445), 84d25d42fd184fd0a93d20554c1e3c20 (127.3.245.1:36053)
I20260620 07:11:33.238186 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.234713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:45951 (local address 127.3.245.1:36053)
0620 07:11:33.234836 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.234842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.234856 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:11:33.234924 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.234927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.234978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.235064 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.235331 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.236012 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.237198 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.237622 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.237744 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.237807 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.237989 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.238064 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:11:33.238185 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.234565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:36053 (local address 127.3.245.3:45951)
0620 07:11:33.234684 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.234700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.234796 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.235079 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.235082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.235099 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.235215 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.235220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.236125 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.236130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.237074 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.237084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.237747 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.237750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.237853 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.237881 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.237971 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.238050 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 07:11:33.238554 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.234673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47653 (local address 127.3.245.2:44445)
0620 07:11:33.234815 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.234823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.235152 (+ 329us) server_negotiation.cc:407] Connection header received
0620 07:11:33.235300 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.235303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.235355 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.235433 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:33.235719 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.236329 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.237547 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.237957 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.238081 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.238136 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.238388 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.238467 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:11:33.238584 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.234565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44445 (local address 127.3.245.3:47653)
0620 07:11:33.235051 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:33.235062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:33.235188 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:33.235454 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:33.235457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:33.235500 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:33.235610 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.235615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.236439 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.236444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.237417 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:33.237427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.238095 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.238098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:33.238195 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.238215 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:11:33.238354 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 07:11:33.238430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":119,"thread_start_us":133,"threads_started":1}
I20260620 07:11:33.239161 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df0b515b048418a0f4df3d3124d08" candidate_uuid: "bea8eb3f29f8426daecebe6333ebd524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1676e77a4f14469992b54f1979117546" is_pre_election: true
I20260620 07:11:33.239171 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df0b515b048418a0f4df3d3124d08" candidate_uuid: "bea8eb3f29f8426daecebe6333ebd524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d25d42fd184fd0a93d20554c1e3c20" is_pre_election: true
I20260620 07:11:33.239467 4767 raft_consensus.cc:2454] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bea8eb3f29f8426daecebe6333ebd524 in term 0.
I20260620 07:11:33.239477 4901 raft_consensus.cc:2454] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bea8eb3f29f8426daecebe6333ebd524 in term 0.
I20260620 07:11:33.239964 4970 leader_election.cc:302] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d25d42fd184fd0a93d20554c1e3c20, bea8eb3f29f8426daecebe6333ebd524; no voters:
I20260620 07:11:33.240259 5104 raft_consensus.cc:2781] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:33.240345 5104 raft_consensus.cc:482] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:33.240386 5104 raft_consensus.cc:3037] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:33.241441 5104 raft_consensus.cc:504] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.241787 5104 leader_election.cc:288] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [CANDIDATE]: Term 1 election: Requested vote from peers 1676e77a4f14469992b54f1979117546 (127.3.245.2:44445), 84d25d42fd184fd0a93d20554c1e3c20 (127.3.245.1:36053)
I20260620 07:11:33.242185 4767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df0b515b048418a0f4df3d3124d08" candidate_uuid: "bea8eb3f29f8426daecebe6333ebd524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d25d42fd184fd0a93d20554c1e3c20"
I20260620 07:11:33.242223 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19df0b515b048418a0f4df3d3124d08" candidate_uuid: "bea8eb3f29f8426daecebe6333ebd524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1676e77a4f14469992b54f1979117546"
I20260620 07:11:33.242300 4767 raft_consensus.cc:3037] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:33.242388 4901 raft_consensus.cc:3037] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:33.243393 4767 raft_consensus.cc:2454] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea8eb3f29f8426daecebe6333ebd524 in term 1.
I20260620 07:11:33.243736 4901 raft_consensus.cc:2454] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bea8eb3f29f8426daecebe6333ebd524 in term 1.
I20260620 07:11:33.243752 4970 leader_election.cc:302] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d25d42fd184fd0a93d20554c1e3c20, bea8eb3f29f8426daecebe6333ebd524; no voters:
I20260620 07:11:33.243961 5104 raft_consensus.cc:2781] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:33.244240 5104 raft_consensus.cc:686] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [term 1 LEADER]: Becoming Leader. State: Replica: bea8eb3f29f8426daecebe6333ebd524, State: Running, Role: LEADER
I20260620 07:11:33.244705 5104 consensus_queue.cc:237] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } }
I20260620 07:11:33.248210 4622 catalog_manager.cc:5495] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524 reported cstate change: term changed from 0 to 1, leader changed from <none> to bea8eb3f29f8426daecebe6333ebd524 (127.3.245.3). New cstate: current_term: 1 leader_uuid: "bea8eb3f29f8426daecebe6333ebd524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea8eb3f29f8426daecebe6333ebd524" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1676e77a4f14469992b54f1979117546" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84d25d42fd184fd0a93d20554c1e3c20" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36053 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:11:33.267827 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.257976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.3.245.1:36053)
0620 07:11:33.258206 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.258210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.258307 (+ 97us) server_negotiation.cc:407] Connection header received
0620 07:11:33.258397 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.258401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.258448 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.258517 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.259433 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.260046 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.261237 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.261467 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.261582 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.264153 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.264167 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.264180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.264209 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.266008 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.266495 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.266501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.266502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.266586 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.266871 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.266875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.266879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.267141 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.267353 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.267521 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.267663 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
W20260620 07:11:33.294082 4948 tablet.cc:2369] T b19df0b515b048418a0f4df3d3124d08 P 1676e77a4f14469992b54f1979117546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:33.296142 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.286671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.3.245.1:36053)
0620 07:11:33.286844 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.286848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.286864 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:11:33.287131 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.287134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.287191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.287271 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.288098 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.288767 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.289787 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.290039 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.290132 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.292738 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.292741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.292744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.292778 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.294395 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.294892 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.294895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.294896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.295013 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.295267 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.295269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.295270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.295437 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.295550 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.295825 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.295927 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 07:11:33.296231 5082 tablet.cc:2369] T b19df0b515b048418a0f4df3d3124d08 P bea8eb3f29f8426daecebe6333ebd524: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[4577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
W20260620 07:11:33.319343 4814 tablet.cc:2369] T b19df0b515b048418a0f4df3d3124d08 P 84d25d42fd184fd0a93d20554c1e3c20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:33.323063 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:33.313972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.3.245.1:36053)
0620 07:11:33.314155 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:33.314160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:33.314289 (+ 129us) server_negotiation.cc:407] Connection header received
0620 07:11:33.314496 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:33.314502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:33.314555 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:33.314643 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:33.315361 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.315899 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.316942 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:33.317166 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:33.317279 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:33.319889 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:33.319894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:33.319896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:33.319935 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:33.321435 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.321902 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.321905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.321906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.321987 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:33.322319 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:33.322322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:33.322323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:33.322505 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:33.322615 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:33.322829 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:11:33.322941 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:11:33.324501 4727 tablet_service.cc:2908] Rejecting scan request for tablet b19df0b515b048418a0f4df3d3124d08: Uninitialized: clean time has not yet been initialized
I20260620 07:11:33.326835 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4684
I20260620 07:11:33.333556 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4818
I20260620 07:11:33.339440 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4952
I20260620 07:11:33.345075 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 4590
2026-06-20T07:11:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2494 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[5122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[5122](info): set up 2 sockets
Jun 20 07:11:33 dist-test-slave-ldfr krb5kdc[5122](info): commencing operation
krb5kdc: starting...
W20260620 07:11:34.428561 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:34 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:34.585083 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36563
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:34.706334 5135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:34.706702 5135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:34.706792 5135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:34.711099 5135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:34.711216 5135 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:11:34.711249 5135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:34.711297 5135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:34.711334 5135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:34.711366 5135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:34.717010 5135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36563
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:34.718222 5135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:34.719343 5135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:34.728592 5143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:34.728798 5135 server_base.cc:1034] running on GCE node
W20260620 07:11:34.728667 5141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:34.728592 5140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:34.729574 5135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:34.730751 5135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:34.731951 5135 hybrid_clock.cc:648] HybridClock initialized: now 1781939494731916 us; error 48 us; skew 500 ppm
Jun 20 07:11:34 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:34.735013 5135 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:34.736054 5135 webserver.cc:460] Webserver started at http://127.3.245.62:40643/ using document root <none> and password file <none>
I20260620 07:11:34.736711 5135 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:34.736809 5135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:34.737058 5135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:34.739563 5135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "643464ffbaff42fa88e12839fe2c4fba"
format_stamp: "Formatted at 2026-06-20 07:11:34 on dist-test-slave-ldfr"
server_key: "e962de19710339dfa548695616c70530"
server_key_iv: "d65b707ed00d2b6ad22e435d7259321d"
server_key_version: "encryptionkey@0"
I20260620 07:11:34.740490 5135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "643464ffbaff42fa88e12839fe2c4fba"
format_stamp: "Formatted at 2026-06-20 07:11:34 on dist-test-slave-ldfr"
server_key: "e962de19710339dfa548695616c70530"
server_key_iv: "d65b707ed00d2b6ad22e435d7259321d"
server_key_version: "encryptionkey@0"
I20260620 07:11:34.744813 5135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:11:34.747607 5150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:34.749185 5135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:34.749372 5135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "643464ffbaff42fa88e12839fe2c4fba"
format_stamp: "Formatted at 2026-06-20 07:11:34 on dist-test-slave-ldfr"
server_key: "e962de19710339dfa548695616c70530"
server_key_iv: "d65b707ed00d2b6ad22e435d7259321d"
server_key_version: "encryptionkey@0"
I20260620 07:11:34.749522 5135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:34.767020 5135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:34.770754 5135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:34.770985 5135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:34.780130 5135 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:36563
I20260620 07:11:34.780145 5212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:36563 every 8 connection(s)
I20260620 07:11:34.781260 5135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:34.782605 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5135
I20260620 07:11:34.782701 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:34.782995 4052 external_mini_cluster.cc:1428] Setting key c348f4335b2913f58f62437c3ced2f1a
I20260620 07:11:34.784504 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:34.791052 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Bootstrap starting.
Jun 20 07:11:34 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:34.793959 5213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:34.795074 5213 log.cc:824] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:34.797808 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: No bootstrap required, opened a new log
I20260620 07:11:34.800566 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:34.784455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.3.245.62:36563)
0620 07:11:34.784937 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:34.784955 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:34.784992 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:11:34.785642 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:34.785659 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:34.786053 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:34.786512 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:34.787337 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.788374 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.789336 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:34.789851 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:34.790041 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:34.793171 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:34.793176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:34.793191 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:34.793224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:34.796463 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.797847 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.797900 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.797907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.798075 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:34.798459 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:34.798463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:34.798465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:34.798924 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:34.799137 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:34.799472 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 07:11:34.799804 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":156,"threads_started":1}
I20260620 07:11:34.801601 5213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } }
I20260620 07:11:34.801913 5213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:34.801976 5213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643464ffbaff42fa88e12839fe2c4fba, State: Initialized, Role: FOLLOWER
I20260620 07:11:34.802559 5213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } }
I20260620 07:11:34.803048 5213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:34.803169 5213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:34.803295 5213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:34.804322 5213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } }
I20260620 07:11:34.804781 5213 leader_election.cc:302] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643464ffbaff42fa88e12839fe2c4fba; no voters:
I20260620 07:11:34.805143 5213 leader_election.cc:288] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:34.805380 5218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:34.805753 5218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [term 1 LEADER]: Becoming Leader. State: Replica: 643464ffbaff42fa88e12839fe2c4fba, State: Running, Role: LEADER
I20260620 07:11:34.806104 5218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } }
I20260620 07:11:34.806484 5213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:34.808000 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643464ffbaff42fa88e12839fe2c4fba. Latest consensus state: current_term: 1 leader_uuid: "643464ffbaff42fa88e12839fe2c4fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } } }
I20260620 07:11:34.808061 5219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643464ffbaff42fa88e12839fe2c4fba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643464ffbaff42fa88e12839fe2c4fba" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36563 } } }
I20260620 07:11:34.808149 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:34.808171 5219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:34.808593 5227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:34.811765 5227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:34.820199 5227 catalog_manager.cc:1347] Generated new cluster ID: c8d5266b1f8a4497a82f104e78dd5adc
I20260620 07:11:34.820318 5227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:34.839601 5227 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:34.840768 5227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:34.848495 5227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Generated new TSK 0
I20260620 07:11:34.849344 5227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:34.888410 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36563
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:35.021281 5239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:35.021884 5239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:35.021970 5239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:35.029587 5239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:35.029680 5239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:35.029739 5239 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:35.029794 5239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:35.035873 5239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:36563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:35.037384 5239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:35.038545 5239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:35.049577 5247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:35.049492 5245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:35.049527 5244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:35.049837 5239 server_base.cc:1034] running on GCE node
I20260620 07:11:35.050362 5239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:35.051064 5239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:35.052238 5239 hybrid_clock.cc:648] HybridClock initialized: now 1781939495052218 us; error 31 us; skew 500 ppm
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:35.059193 5239 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:35.060228 5239 webserver.cc:460] Webserver started at http://127.3.245.1:35275/ using document root <none> and password file <none>
I20260620 07:11:35.060932 5239 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:35.061028 5239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:35.061264 5239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:35.063432 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "bb38349d5a72479cab496909bc00ae43"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "fa789261ece1db259b87372142bf9918"
server_key_iv: "03b2dd79f042d05cc24732a19113caf2"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.064013 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "bb38349d5a72479cab496909bc00ae43"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "fa789261ece1db259b87372142bf9918"
server_key_iv: "03b2dd79f042d05cc24732a19113caf2"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.068357 5239 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:35.071319 5254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.072885 5239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:35.073052 5239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "bb38349d5a72479cab496909bc00ae43"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "fa789261ece1db259b87372142bf9918"
server_key_iv: "03b2dd79f042d05cc24732a19113caf2"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.073186 5239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:35.089349 5239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:35.093029 5239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:35.093252 5239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:35.094836 5239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:35.094913 5239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.094986 5239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:35.095028 5239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:35.106397 5239 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:36877
I20260620 07:11:35.106415 5369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:36877 every 8 connection(s)
I20260620 07:11:35.107725 5239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:35.110810 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.095648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:39171 (local address 127.3.245.62:36563)
0620 07:11:35.095768 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.095772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.096972 (+ 1200us) server_negotiation.cc:407] Connection header received
0620 07:11:35.098179 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.098182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.098256 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.098366 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.099783 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.100691 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.102095 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.102559 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.102701 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.106479 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.106485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.106488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.106521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.108645 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.109236 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.109241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.109242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.109335 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.109688 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.109693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.109695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.109901 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.110010 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.110507 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.110642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:11:35.111678 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.096212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.1:39171)
0620 07:11:35.096812 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.096857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.097819 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.098567 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.098574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.098937 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.099612 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.099636 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.100825 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.100829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.101937 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.101948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.102748 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.102754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.102998 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.104282 (+ 1284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.104301 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.106231 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.108774 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.108780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.108794 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.109125 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.109437 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.109441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.109443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.109590 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.110008 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.110015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.110269 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.110463 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.110807 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":128,"threads_started":1}
I20260620 07:11:35.113111 5370 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36563
I20260620 07:11:35.113386 5370 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:35.113921 5370 heartbeater.cc:511] Master 127.3.245.62:36563 requested a full tablet report, sending...
I20260620 07:11:35.115681 5162 ts_manager.cc:194] Registered new tserver with Master: bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877)
I20260620 07:11:35.117110 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5239
I20260620 07:11:35.117156 5162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:39171
I20260620 07:11:35.117240 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:35.117509 4052 external_mini_cluster.cc:1428] Setting key d052b84bc6cbf10fb1ad1d0b6895b332
I20260620 07:11:35.123991 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.115540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:56509 (local address 127.3.245.62:36563)
0620 07:11:35.115677 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.115684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.115848 (+ 164us) server_negotiation.cc:407] Connection header received
0620 07:11:35.116020 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.116022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.116062 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.116151 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.116729 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.117526 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.118592 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.118911 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.119032 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.120492 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.120495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.120498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.120524 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.122066 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.122587 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.122590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.122591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.122683 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.123035 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.123050 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.123051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.123243 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.123385 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.123677 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.123797 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:35.123991 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.115472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.1:56509)
0620 07:11:35.115763 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.115780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.115917 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.116170 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.116173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.116395 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.116524 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.116532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.117702 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.117705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.118469 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.118478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.119159 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.119163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.119256 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.119827 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.119848 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.120339 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.122174 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.122180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.122185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.122467 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.122786 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.122792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.122797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.122904 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.123413 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.123416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.123507 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.123622 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.123782 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:35.148474 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36563
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:35.273401 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:35.273752 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:35.273838 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:35.278362 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:35.278460 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:35.278542 5379 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:35.278626 5379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:35.284372 5379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:36563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:35.285575 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:35.286641 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:35.296996 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:35.297143 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:35.297269 5379 server_base.cc:1034] running on GCE node
W20260620 07:11:35.297143 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:35.297827 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:35.298506 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:35.299727 5379 hybrid_clock.cc:648] HybridClock initialized: now 1781939495299695 us; error 54 us; skew 500 ppm
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:35.302872 5379 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:35.303879 5379 webserver.cc:460] Webserver started at http://127.3.245.2:34249/ using document root <none> and password file <none>
I20260620 07:11:35.304513 5379 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:35.304612 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:35.304888 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:35.307174 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "4808a6921503491ba8c0c7636997ebe9"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c2aebb306101373c0ec135c36e112526"
server_key_iv: "967450d5d44a2552cbf880d1afd1660b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.307824 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "4808a6921503491ba8c0c7636997ebe9"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c2aebb306101373c0ec135c36e112526"
server_key_iv: "967450d5d44a2552cbf880d1afd1660b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.311918 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:35.314733 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.316114 5379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:35.316265 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "4808a6921503491ba8c0c7636997ebe9"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c2aebb306101373c0ec135c36e112526"
server_key_iv: "967450d5d44a2552cbf880d1afd1660b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.316392 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:35.328239 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:35.331899 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:35.332150 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:35.333771 5379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:35.333849 5379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.333931 5379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:35.333984 5379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:35.345860 5379 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:37759
I20260620 07:11:35.347087 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:35.347729 5509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:37759 every 8 connection(s)
I20260620 07:11:35.349079 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.334638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:35739 (local address 127.3.245.62:36563)
0620 07:11:35.334776 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.334779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.335997 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 07:11:35.337297 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.337305 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.337383 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.337507 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.338910 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.339579 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.340907 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.341149 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.341274 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.344336 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.344340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.344343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.344377 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.346310 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.346909 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.346914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.346916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.347033 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.347557 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.347564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.347566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.347913 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.348038 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.348782 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.348956 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:11:35.349833 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.335204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.2:35739)
0620 07:11:35.335830 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.335873 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.336933 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.337716 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.337724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.338132 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.338737 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.338758 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.339726 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.339730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.340779 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.340789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.341471 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.341474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.341622 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.342285 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.342312 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.344102 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.346500 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.346506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.346518 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.346784 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.347216 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.347219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.347221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.347382 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.348043 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.348049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.348371 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.348597 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.348949 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":152,"threads_started":1}
I20260620 07:11:35.351579 5510 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36563
I20260620 07:11:35.351766 5510 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:35.352366 5510 heartbeater.cc:511] Master 127.3.245.62:36563 requested a full tablet report, sending...
I20260620 07:11:35.353606 5162 ts_manager.cc:194] Registered new tserver with Master: 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2:37759)
I20260620 07:11:35.354472 5162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:35739
I20260620 07:11:35.356134 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5379
I20260620 07:11:35.356248 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:35.356546 4052 external_mini_cluster.cc:1428] Setting key e884911a4b2b1d1624eb1fe9443b0f0c
I20260620 07:11:35.362798 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.2:35659)
0620 07:11:35.354298 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.354311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.354396 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.354651 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.354653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.354908 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.355015 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.355021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.355820 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.355824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.356858 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.356870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.357465 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.357467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.357539 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.358161 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.358177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.358689 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.361068 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.361071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.361072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.361335 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.361663 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.361668 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.361670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.361758 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.362222 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.362225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.362334 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.362446 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.362608 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":163,"threads_started":1}
I20260620 07:11:35.362815 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.354069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:35659 (local address 127.3.245.62:36563)
0620 07:11:35.354208 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.354212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.354380 (+ 168us) server_negotiation.cc:407] Connection header received
0620 07:11:35.354499 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.354503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.354552 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.354636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.355133 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.355659 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.357004 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.357340 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.357450 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.358943 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.358948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.358954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.358983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.360893 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.361471 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.361477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.361479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.361567 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.361868 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.361872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.361874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.362116 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.362232 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.362497 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.362610 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:35.390223 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36563
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:35.507352 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:35.507839 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:35.507894 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:35.512331 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:35.512410 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:35.512451 5519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:35.512486 5519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:35.518399 5519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:36563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:35.519500 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:35.520644 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:35.531347 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:35.531347 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:35.531347 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:35.531412 5519 server_base.cc:1034] running on GCE node
I20260620 07:11:35.532142 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:35.532900 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:35.534078 5519 hybrid_clock.cc:648] HybridClock initialized: now 1781939495534044 us; error 56 us; skew 500 ppm
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:35.537040 5519 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:35.538168 5519 webserver.cc:460] Webserver started at http://127.3.245.3:36385/ using document root <none> and password file <none>
I20260620 07:11:35.538865 5519 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:35.538942 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:35.539187 5519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:35.541656 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "79b39c7a16d54c198d1c97affb128ffc"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c84ea41a1f0f87743e719363d64cc90b"
server_key_iv: "2a5b5d8e3adf496c67b4392e552f0e1b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.542336 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "79b39c7a16d54c198d1c97affb128ffc"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c84ea41a1f0f87743e719363d64cc90b"
server_key_iv: "2a5b5d8e3adf496c67b4392e552f0e1b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.546869 5519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:11:35.549829 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.551302 5519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:35.551491 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "79b39c7a16d54c198d1c97affb128ffc"
format_stamp: "Formatted at 2026-06-20 07:11:35 on dist-test-slave-ldfr"
server_key: "c84ea41a1f0f87743e719363d64cc90b"
server_key_iv: "2a5b5d8e3adf496c67b4392e552f0e1b"
server_key_version: "encryptionkey@0"
I20260620 07:11:35.551625 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:35.570297 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:35.574057 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:35.574291 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:35.576023 5519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:35.576113 5519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:35.576193 5519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:35.576259 5519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:35.587901 5519 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:42023
I20260620 07:11:35.587996 5649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:42023 every 8 connection(s)
I20260620 07:11:35.589071 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:35.591106 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.576951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:54069 (local address 127.3.245.62:36563)
0620 07:11:35.577107 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.577112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.578177 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 07:11:35.579316 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.579318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.579375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.579421 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.581049 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.581726 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.583047 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.583298 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.583452 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.586823 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.586829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.586833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.586867 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.588703 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.589358 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.589365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.589367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.589489 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.589902 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.589908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.589911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.590163 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.590262 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.590863 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.590975 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:11:35.591961 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.577540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.3:54069)
0620 07:11:35.578000 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.578051 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.579041 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.579744 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.579750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.580150 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.580899 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.580917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.581853 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.581859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.582913 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.582923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.583498 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.583504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.583774 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.584574 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.584602 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.586590 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.588842 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.588847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.588858 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.589250 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.589615 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.589618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.589620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.589806 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.590298 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.590303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.590553 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.590729 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.591057 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
I20260620 07:11:35.593573 5650 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36563
I20260620 07:11:35.593744 5650 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:35.594307 5650 heartbeater.cc:511] Master 127.3.245.62:36563 requested a full tablet report, sending...
I20260620 07:11:35.595479 5167 ts_manager.cc:194] Registered new tserver with Master: 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023)
I20260620 07:11:35.596344 5167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:54069
I20260620 07:11:35.597996 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5519
I20260620 07:11:35.598131 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:35.598428 4052 external_mini_cluster.cc:1428] Setting key e2648e303525ad5e145bb949fc66e321
I20260620 07:11:35.600732 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:35.604249 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.595709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:45143 (local address 127.3.245.62:36563)
0620 07:11:35.595824 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.595828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.596093 (+ 265us) server_negotiation.cc:407] Connection header received
0620 07:11:35.596201 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.596203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.596241 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.596309 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.596857 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.597620 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.598944 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.599214 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.599364 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.600737 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.600741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.600748 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.600781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.602475 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.602989 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.602993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.602995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.603080 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.603368 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.603373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.603375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.603578 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.603690 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.603956 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.604070 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:11:35.604255 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.595626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.3.245.3:45143)
0620 07:11:35.595998 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.596013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.596108 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.596351 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.596357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.596579 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.596747 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.596755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.597787 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.597793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.598801 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.598811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.599350 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.599353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.599439 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.600072 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.600087 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.600556 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.602591 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.602593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.602595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.602805 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.603178 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.603182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.603185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.603273 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.603678 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.603681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.603786 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.603895 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.604055 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":91,"threads_started":1}
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:35.629462 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.620140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.3.245.62:36563)
0620 07:11:35.620381 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.620387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.620413 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:11:35.620460 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.620462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.620507 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.620587 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.621494 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.622092 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.623007 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.623246 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.623347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.626084 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.626087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.626089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.626116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.627644 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.628112 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.628115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.628116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.628196 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.628490 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.628492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.628493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.628757 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.628860 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.629135 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.629242 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:11:35.632603 5167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:35.635636 5167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:35.651100 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.645007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.3.245.3:42023)
0620 07:11:35.645423 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.645435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.645647 (+ 212us) server_negotiation.cc:407] Connection header received
0620 07:11:35.646390 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.646396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.646577 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.646704 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.647140 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.648191 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.649713 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650326 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650455 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.650632 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.650818 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.650926 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":110,"threads_started":1}
I20260620 07:11:35.651280 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.644387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:36877 (local address 127.0.0.1:48002)
0620 07:11:35.645353 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.645409 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.645537 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.646344 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.646351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.646383 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.646535 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.646542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.647841 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.647845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.649516 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.649527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650771 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.650910 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.650938 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.651006 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.651137 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":122,"thread_start_us":111,"threads_started":1}
I20260620 07:11:35.651531 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.644418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42023 (local address 127.0.0.1:60108)
0620 07:11:35.645558 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.645571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.646184 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.646898 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.646901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.646922 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.647013 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.647018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.648329 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.648333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.649574 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.649584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650462 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.650601 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.650647 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.651356 (+ 709us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.651432 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260620 07:11:35.651827 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.645233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.3.245.2:37759)
0620 07:11:35.645655 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.645667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.646017 (+ 350us) server_negotiation.cc:407] Connection header received
0620 07:11:35.646194 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.646199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.646358 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.646483 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.646787 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.647677 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.649454 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650441 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650610 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.650783 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.651621 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.651718 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":125,"threads_started":1}
I20260620 07:11:35.651939 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.644695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37759 (local address 127.0.0.1:40934)
0620 07:11:35.645783 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.645797 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.646059 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.646524 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.646527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.646544 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.646664 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.646670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.647791 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.647798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.649333 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.649342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650857 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.651023 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.651098 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.651562 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.651735 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":549,"thread_start_us":101,"threads_started":1}
I20260620 07:11:35.652133 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.644666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.3.245.1:36877)
0620 07:11:35.645827 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.645833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.645858 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:11:35.645937 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.645941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.646191 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.646350 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.646644 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.647696 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.649650 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.650643 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.650776 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.650916 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.651084 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.651243 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":81,"threads_started":1}
I20260620 07:11:35.654901 5584 tablet_service.cc:1467] Processing CreateTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 (DEFAULT_TABLE table=test-table [id=605fde7665454e08b7005fb87f06adc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.654887 5304 tablet_service.cc:1467] Processing CreateTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 (DEFAULT_TABLE table=test-table [id=605fde7665454e08b7005fb87f06adc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.655946 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c275bb3af3a401a84234aaf0b42a564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.655957 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c275bb3af3a401a84234aaf0b42a564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.658892 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 (DEFAULT_TABLE table=test-table [id=605fde7665454e08b7005fb87f06adc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.660027 5444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c275bb3af3a401a84234aaf0b42a564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.663044 5673 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Bootstrap starting.
I20260620 07:11:35.664359 5674 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Bootstrap starting.
I20260620 07:11:35.666501 5673 tablet_bootstrap.cc:654] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.666579 5674 tablet_bootstrap.cc:654] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.667433 5674 log.cc:824] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:35.667433 5673 log.cc:824] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:35.667475 5675 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Bootstrap starting.
I20260620 07:11:35.669495 5674 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: No bootstrap required, opened a new log
I20260620 07:11:35.669498 5673 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: No bootstrap required, opened a new log
I20260620 07:11:35.669641 5675 tablet_bootstrap.cc:654] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.669701 5674 ts_tablet_manager.cc:1397] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:35.669701 5673 ts_tablet_manager.cc:1397] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:11:35.670360 5675 log.cc:824] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:35.672441 5675 tablet_bootstrap.cc:492] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: No bootstrap required, opened a new log
I20260620 07:11:35.672722 5675 ts_tablet_manager.cc:1397] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:11:35.674368 5674 raft_consensus.cc:348] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.674368 5673 raft_consensus.cc:348] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.674659 5673 raft_consensus.cc:374] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.674659 5674 raft_consensus.cc:374] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.674713 5673 raft_consensus.cc:729] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b39c7a16d54c198d1c97affb128ffc, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.674722 5674 raft_consensus.cc:729] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb38349d5a72479cab496909bc00ae43, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.675256 5674 consensus_queue.cc:260] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.675256 5673 consensus_queue.cc:260] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.675976 5370 heartbeater.cc:503] Master 127.3.245.62:36563 was elected leader, sending a full tablet report...
I20260620 07:11:35.676184 5675 raft_consensus.cc:348] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.676326 5673 ts_tablet_manager.cc:1428] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 07:11:35.676425 5675 raft_consensus.cc:374] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.676487 5675 raft_consensus.cc:729] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4808a6921503491ba8c0c7636997ebe9, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.676759 5650 heartbeater.cc:503] Master 127.3.245.62:36563 was elected leader, sending a full tablet report...
I20260620 07:11:35.676956 5675 consensus_queue.cc:260] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.677343 5674 ts_tablet_manager.cc:1428] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:11:35.678026 5510 heartbeater.cc:503] Master 127.3.245.62:36563 was elected leader, sending a full tablet report...
I20260620 07:11:35.678841 5675 ts_tablet_manager.cc:1428] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:11:35.713068 5679 raft_consensus.cc:482] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:35.713321 5679 raft_consensus.cc:504] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.714529 5679 leader_election.cc:288] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023), 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2:37759)
I20260620 07:11:35.718712 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.714909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:49255 (local address 127.3.245.2:37759)
0620 07:11:35.715055 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.715060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.715344 (+ 284us) server_negotiation.cc:407] Connection header received
0620 07:11:35.715546 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.715549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.715592 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.715679 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.716057 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.716556 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.717646 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.718113 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.718229 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.718292 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.718518 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.718587 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:11:35.718717 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.715023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:57207 (local address 127.3.245.3:42023)
0620 07:11:35.715145 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.715149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.715332 (+ 183us) server_negotiation.cc:407] Connection header received
0620 07:11:35.715523 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.715526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.715577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.715661 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.715979 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.716493 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.717602 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.718130 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.718246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.718327 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.718525 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.718587 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:11:35.718776 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.714807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42023 (local address 127.3.245.1:57207)
0620 07:11:35.715229 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.715241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.715408 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.715674 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.715676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.715692 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.715874 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.715880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.716603 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.716629 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.717483 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.717491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.718247 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.718251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.718406 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.718429 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.718511 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.718582 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":44,"spinlock_wait_cycles":33792,"thread_start_us":157,"threads_started":1}
I20260620 07:11:35.718809 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.714807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37759 (local address 127.3.245.1:49255)
0620 07:11:35.715244 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.715256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.715411 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.715690 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.715693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.715708 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.715884 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.715889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.716724 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.716729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.717519 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.717527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.718234 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.718237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.718394 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.718412 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.718503 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.718574 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":204,"spinlock_wait_cycles":2176,"thread_start_us":90,"threads_started":1}
I20260620 07:11:35.719434 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c275bb3af3a401a84234aaf0b42a564" candidate_uuid: "bb38349d5a72479cab496909bc00ae43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4808a6921503491ba8c0c7636997ebe9" is_pre_election: true
I20260620 07:11:35.719749 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c275bb3af3a401a84234aaf0b42a564" candidate_uuid: "bb38349d5a72479cab496909bc00ae43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc" is_pre_election: true
I20260620 07:11:35.719810 5464 raft_consensus.cc:2454] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb38349d5a72479cab496909bc00ae43 in term 0.
I20260620 07:11:35.720027 5604 raft_consensus.cc:2454] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb38349d5a72479cab496909bc00ae43 in term 0.
I20260620 07:11:35.720526 5255 leader_election.cc:302] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4808a6921503491ba8c0c7636997ebe9, 79b39c7a16d54c198d1c97affb128ffc, bb38349d5a72479cab496909bc00ae43; no voters:
I20260620 07:11:35.720927 5679 raft_consensus.cc:2781] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:35.721006 5679 raft_consensus.cc:482] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:35.721176 5679 raft_consensus.cc:3037] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.722237 5679 raft_consensus.cc:504] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.722585 5679 leader_election.cc:288] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [CANDIDATE]: Term 1 election: Requested vote from peers 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023), 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2:37759)
I20260620 07:11:35.722956 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c275bb3af3a401a84234aaf0b42a564" candidate_uuid: "bb38349d5a72479cab496909bc00ae43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc"
I20260620 07:11:35.723063 5604 raft_consensus.cc:3037] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.724206 5604 raft_consensus.cc:2454] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb38349d5a72479cab496909bc00ae43 in term 1.
I20260620 07:11:35.724485 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c275bb3af3a401a84234aaf0b42a564" candidate_uuid: "bb38349d5a72479cab496909bc00ae43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4808a6921503491ba8c0c7636997ebe9"
I20260620 07:11:35.724582 5256 leader_election.cc:302] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b39c7a16d54c198d1c97affb128ffc, bb38349d5a72479cab496909bc00ae43; no voters:
I20260620 07:11:35.724668 5464 raft_consensus.cc:3037] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.724839 5679 raft_consensus.cc:2781] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:35.725075 5679 raft_consensus.cc:686] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 1 LEADER]: Becoming Leader. State: Replica: bb38349d5a72479cab496909bc00ae43, State: Running, Role: LEADER
I20260620 07:11:35.725458 5679 consensus_queue.cc:237] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } }
I20260620 07:11:35.726074 5464 raft_consensus.cc:2454] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb38349d5a72479cab496909bc00ae43 in term 1.
I20260620 07:11:35.729676 5163 catalog_manager.cc:5495] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb38349d5a72479cab496909bc00ae43 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "bb38349d5a72479cab496909bc00ae43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:35.761292 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.757613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.3.245.1:36877)
0620 07:11:35.757748 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.757752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.757764 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:11:35.757835 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.757837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.757892 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.757993 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:35.758359 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.758931 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.759793 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.760040 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.760132 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.760408 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:35.760876 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:35.760970 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.761118 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.761174 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:11:35.767452 5464 raft_consensus.cc:1261] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Refusing update from remote peer bb38349d5a72479cab496909bc00ae43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:35.767834 5604 raft_consensus.cc:1261] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Refusing update from remote peer bb38349d5a72479cab496909bc00ae43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:35.768381 5679 consensus_queue.cc:1035] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:35.768587 5684 consensus_queue.cc:1035] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:35.777714 5689 mvcc.cc:205] Tried to move back new op lower bound from 7298824174655893504 to 7298824174495965184. Current Snapshot: MvccSnapshot[applied={T|T < 7298824174655893504}]
I20260620 07:11:35.778405 5688 mvcc.cc:205] Tried to move back new op lower bound from 7298824174655893504 to 7298824174495965184. Current Snapshot: MvccSnapshot[applied={T|T < 7298824174655893504}]
I20260620 07:11:35.796981 5167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:35.797179 5167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:35.799698 5167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Sending DeleteTablet for 3 replicas of tablet 9c275bb3af3a401a84234aaf0b42a564
I20260620 07:11:35.800534 5304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:48002
I20260620 07:11:35.800570 5584 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60108
I20260620 07:11:35.800577 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c275bb3af3a401a84234aaf0b42a564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:40934
I20260620 07:11:35.801097 5697 tablet_replica.cc:330] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: stopping tablet replica
I20260620 07:11:35.801297 5698 tablet_replica.cc:330] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: stopping tablet replica
I20260620 07:11:35.801424 5697 raft_consensus.cc:2229] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:35.801614 5698 raft_consensus.cc:2229] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:35.801843 5697 raft_consensus.cc:2258] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:35.801990 5698 raft_consensus.cc:2258] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:35.802335 5167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:35.802767 5167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:35.802948 5699 tablet_replica.cc:330] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: stopping tablet replica
I20260620 07:11:35.803241 5699 raft_consensus.cc:2229] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:35.803568 5699 raft_consensus.cc:2258] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:35.804080 5698 ts_tablet_manager.cc:1905] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:35.804728 5697 ts_tablet_manager.cc:1905] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:35.807415 5699 ts_tablet_manager.cc:1905] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:35.808126 5697 ts_tablet_manager.cc:1918] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:35.808265 5697 log.cc:1196] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/9c275bb3af3a401a84234aaf0b42a564
I20260620 07:11:35.808789 5304 tablet_service.cc:1467] Processing CreateTablet for tablet 34b213835ccf40b2a64a141a8f7842bf (DEFAULT_TABLE table=test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.809015 5697 ts_tablet_manager.cc:1939] T 9c275bb3af3a401a84234aaf0b42a564 P bb38349d5a72479cab496909bc00ae43: Deleting consensus metadata
I20260620 07:11:35.809058 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b213835ccf40b2a64a141a8f7842bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.809180 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 34b213835ccf40b2a64a141a8f7842bf (DEFAULT_TABLE table=test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.809470 5444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b213835ccf40b2a64a141a8f7842bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.811192 5674 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Bootstrap starting.
I20260620 07:11:35.811534 5154 catalog_manager.cc:4841] TS bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877): tablet 9c275bb3af3a401a84234aaf0b42a564 (table test-table [id=605fde7665454e08b7005fb87f06adc8]) successfully deleted
I20260620 07:11:35.812510 5675 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Bootstrap starting.
I20260620 07:11:35.812705 5674 tablet_bootstrap.cc:654] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.812716 5584 tablet_service.cc:1467] Processing CreateTablet for tablet 34b213835ccf40b2a64a141a8f7842bf (DEFAULT_TABLE table=test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:35.813033 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b213835ccf40b2a64a141a8f7842bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.813766 5675 tablet_bootstrap.cc:654] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.814498 5698 ts_tablet_manager.cc:1918] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:35.815124 5699 ts_tablet_manager.cc:1918] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:35.815286 5673 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Bootstrap starting.
I20260620 07:11:35.815374 5699 log.cc:1196] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/9c275bb3af3a401a84234aaf0b42a564
I20260620 07:11:35.815730 5699 ts_tablet_manager.cc:1939] T 9c275bb3af3a401a84234aaf0b42a564 P 4808a6921503491ba8c0c7636997ebe9: Deleting consensus metadata
I20260620 07:11:35.816013 5674 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: No bootstrap required, opened a new log
I20260620 07:11:35.816118 5674 ts_tablet_manager.cc:1397] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 07:11:35.816341 5673 tablet_bootstrap.cc:654] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.815285 5698 log.cc:1196] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/9c275bb3af3a401a84234aaf0b42a564
I20260620 07:11:35.816630 5674 raft_consensus.cc:348] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.816751 5674 raft_consensus.cc:374] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.816799 5674 raft_consensus.cc:729] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb38349d5a72479cab496909bc00ae43, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.816932 5674 consensus_queue.cc:260] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.817129 5698 ts_tablet_manager.cc:1939] T 9c275bb3af3a401a84234aaf0b42a564 P 79b39c7a16d54c198d1c97affb128ffc: Deleting consensus metadata
I20260620 07:11:35.817188 5674 ts_tablet_manager.cc:1428] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 07:11:35.817454 5151 catalog_manager.cc:4841] TS 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2:37759): tablet 9c275bb3af3a401a84234aaf0b42a564 (table test-table [id=605fde7665454e08b7005fb87f06adc8]) successfully deleted
I20260620 07:11:35.818361 5673 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: No bootstrap required, opened a new log
I20260620 07:11:35.818463 5673 ts_tablet_manager.cc:1397] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:35.818737 5675 tablet_bootstrap.cc:492] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: No bootstrap required, opened a new log
I20260620 07:11:35.818831 5675 ts_tablet_manager.cc:1397] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 07:11:35.818920 5673 raft_consensus.cc:348] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.819048 5673 raft_consensus.cc:374] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.819087 5673 raft_consensus.cc:729] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b39c7a16d54c198d1c97affb128ffc, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.819226 5673 consensus_queue.cc:260] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.819332 5152 catalog_manager.cc:4841] TS 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023): tablet 9c275bb3af3a401a84234aaf0b42a564 (table test-table [id=605fde7665454e08b7005fb87f06adc8]) successfully deleted
I20260620 07:11:35.819340 5675 raft_consensus.cc:348] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.819449 5675 raft_consensus.cc:374] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.819494 5675 raft_consensus.cc:729] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4808a6921503491ba8c0c7636997ebe9, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.819516 5673 ts_tablet_manager.cc:1428] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:35.819619 5675 consensus_queue.cc:260] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.819895 5675 ts_tablet_manager.cc:1428] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260620 07:11:35.840090 5651 tablet.cc:2369] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:35.848368 5511 tablet.cc:2369] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:35.859489 5371 tablet.cc:2369] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:35.867681 5681 raft_consensus.cc:482] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:35.867834 5681 raft_consensus.cc:504] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.869158 5681 leader_election.cc:288] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877), 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023)
I20260620 07:11:35.873955 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.869460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:55949 (local address 127.3.245.1:36877)
0620 07:11:35.869596 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.869601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.869781 (+ 180us) server_negotiation.cc:407] Connection header received
0620 07:11:35.870014 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.870019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.870073 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.870123 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.870494 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.871284 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.872903 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.873332 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.873464 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.873535 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.873767 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.873842 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:11:35.873976 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.869347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:36877 (local address 127.3.245.2:55949)
0620 07:11:35.869679 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.869694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.869878 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.870283 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.870286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.870297 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.870365 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.870369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.871491 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.871494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.872766 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.872777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.873466 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.873469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.873600 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.873633 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.873721 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.873804 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":46,"spinlock_wait_cycles":23680,"thread_start_us":96,"threads_started":1}
I20260620 07:11:35.874522 5324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b213835ccf40b2a64a141a8f7842bf" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb38349d5a72479cab496909bc00ae43" is_pre_election: true
I20260620 07:11:35.874864 5324 raft_consensus.cc:2454] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 0.
I20260620 07:11:35.875082 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.869427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42023 (local address 127.3.245.2:38715)
0620 07:11:35.869750 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.869763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.869878 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.870208 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.870211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.870234 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:35.870351 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.870358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.872357 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.872360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.873635 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.873647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.874656 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.874659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.874767 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.874789 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.874862 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.874929 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":9,"spinlock_wait_cycles":9984,"thread_start_us":74,"threads_started":1}
I20260620 07:11:35.875105 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.869837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:38715 (local address 127.3.245.3:42023)
0620 07:11:35.869955 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.869959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.869974 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:11:35.870030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.870033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.870080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.870166 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:35.871460 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.872218 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.873791 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.874405 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.874500 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.874561 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.874901 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.874982 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:11:35.875329 5398 leader_election.cc:302] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4808a6921503491ba8c0c7636997ebe9, bb38349d5a72479cab496909bc00ae43; no voters:
I20260620 07:11:35.875496 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b213835ccf40b2a64a141a8f7842bf" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc" is_pre_election: true
I20260620 07:11:35.875610 5681 raft_consensus.cc:2781] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:35.875633 5604 raft_consensus.cc:2454] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 0.
I20260620 07:11:35.875689 5681 raft_consensus.cc:482] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:35.875727 5681 raft_consensus.cc:3037] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.877046 5681 raft_consensus.cc:504] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.877477 5681 leader_election.cc:288] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 election: Requested vote from peers bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877), 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023)
I20260620 07:11:35.877820 5324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b213835ccf40b2a64a141a8f7842bf" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb38349d5a72479cab496909bc00ae43"
I20260620 07:11:35.877859 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b213835ccf40b2a64a141a8f7842bf" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc"
I20260620 07:11:35.877959 5324 raft_consensus.cc:3037] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.877980 5604 raft_consensus.cc:3037] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:35.879120 5324 raft_consensus.cc:2454] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 1.
I20260620 07:11:35.879128 5604 raft_consensus.cc:2454] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 1.
I20260620 07:11:35.879455 5398 leader_election.cc:302] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4808a6921503491ba8c0c7636997ebe9, bb38349d5a72479cab496909bc00ae43; no voters:
I20260620 07:11:35.879657 5681 raft_consensus.cc:2781] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:35.879894 5681 raft_consensus.cc:686] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 1 LEADER]: Becoming Leader. State: Replica: 4808a6921503491ba8c0c7636997ebe9, State: Running, Role: LEADER
I20260620 07:11:35.880153 5681 consensus_queue.cc:237] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.882721 5167 catalog_manager.cc:5495] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2). New cstate: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:11:35 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939494, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:35.933218 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.922471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36563 (local address 127.0.0.1:41238)
0620 07:11:35.922787 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:35.922806 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:35.922920 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:35.923190 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:35.923192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:35.923574 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:35.923670 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.923675 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.924763 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.924765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.925322 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:35.925330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.925977 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.925984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:35.926079 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.926800 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:35.926864 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:35.928876 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:35.931448 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.931459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.931465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.931714 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.932074 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:35.932077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:35.932078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:35.932165 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:35.932717 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:35.932723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:35.932834 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 07:11:35.932932 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:11:35.933072 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":118,"threads_started":1}
I20260620 07:11:35.933372 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:35.922589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.3.245.62:36563)
0620 07:11:35.922748 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:35.922754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:35.922888 (+ 134us) server_negotiation.cc:407] Connection header received
0620 07:11:35.922998 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:35.923001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:35.923053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:35.923162 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:35.923801 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.924601 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.925448 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:35.925811 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:35.925926 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:35.929042 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:35.929047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:35.929049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:35.929083 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:35.931253 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.931839 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.931843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.931844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.931942 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:35.932277 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:35.932281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:35.932283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:35.932547 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:35.932682 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:35.933025 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:11:35.933159 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:11:35.937819 5167 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:41238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:11:35.938721 5167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:35.944195 5444 tablet_service.cc:1467] Processing CreateTablet for tablet e2731d3bb2a34f218156d8eabbad5e2d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=950e0b785c2d4b72b80159ed3bb2a548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:35.944195 5304 tablet_service.cc:1467] Processing CreateTablet for tablet e2731d3bb2a34f218156d8eabbad5e2d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=950e0b785c2d4b72b80159ed3bb2a548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:35.944460 5584 tablet_service.cc:1467] Processing CreateTablet for tablet e2731d3bb2a34f218156d8eabbad5e2d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=950e0b785c2d4b72b80159ed3bb2a548]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:35.944643 5444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2731d3bb2a34f218156d8eabbad5e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.944643 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2731d3bb2a34f218156d8eabbad5e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.944875 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2731d3bb2a34f218156d8eabbad5e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:35.947584 5675 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: Bootstrap starting.
I20260620 07:11:35.947912 5673 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: Bootstrap starting.
I20260620 07:11:35.948133 5674 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: Bootstrap starting.
I20260620 07:11:35.948717 5675 tablet_bootstrap.cc:654] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.948920 5673 tablet_bootstrap.cc:654] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.949167 5674 tablet_bootstrap.cc:654] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:35.950649 5675 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: No bootstrap required, opened a new log
I20260620 07:11:35.950747 5675 ts_tablet_manager.cc:1397] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 07:11:35.951227 5675 raft_consensus.cc:348] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.951371 5675 raft_consensus.cc:374] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.951409 5675 raft_consensus.cc:729] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4808a6921503491ba8c0c7636997ebe9, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.951565 5675 consensus_queue.cc:260] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.951622 5673 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: No bootstrap required, opened a new log
I20260620 07:11:35.951731 5673 ts_tablet_manager.cc:1397] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:11:35.951848 5681 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:35.952004 5681 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:35.952203 5673 raft_consensus.cc:348] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.952316 5673 raft_consensus.cc:374] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.952347 5673 raft_consensus.cc:729] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b39c7a16d54c198d1c97affb128ffc, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.952467 5673 consensus_queue.cc:260] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.952767 5673 ts_tablet_manager.cc:1428] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:35.952718 5680 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:35.952872 5680 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:35.954501 5674 tablet_bootstrap.cc:492] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: No bootstrap required, opened a new log
I20260620 07:11:35.954602 5674 ts_tablet_manager.cc:1397] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 07:11:35.955044 5674 raft_consensus.cc:348] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.955132 5674 raft_consensus.cc:374] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:35.955153 5674 raft_consensus.cc:729] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb38349d5a72479cab496909bc00ae43, State: Initialized, Role: FOLLOWER
I20260620 07:11:35.955284 5674 consensus_queue.cc:260] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:35.955574 5674 ts_tablet_manager.cc:1428] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:35.955547 5692 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:35.955688 5692 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:35.951856 5675 ts_tablet_manager.cc:1428] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:36.079366 5681 raft_consensus.cc:482] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:36.079566 5681 raft_consensus.cc:504] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:36.080034 5681 leader_election.cc:288] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877), 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023)
I20260620 07:11:36.080544 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2731d3bb2a34f218156d8eabbad5e2d" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc" is_pre_election: true
I20260620 07:11:36.080574 5324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2731d3bb2a34f218156d8eabbad5e2d" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb38349d5a72479cab496909bc00ae43" is_pre_election: true
I20260620 07:11:36.080761 5324 raft_consensus.cc:2454] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 0.
I20260620 07:11:36.080761 5604 raft_consensus.cc:2454] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 0.
I20260620 07:11:36.081102 5396 leader_election.cc:302] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4808a6921503491ba8c0c7636997ebe9, 79b39c7a16d54c198d1c97affb128ffc; no voters:
I20260620 07:11:36.081323 5681 raft_consensus.cc:2781] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:36.081386 5681 raft_consensus.cc:482] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:36.081415 5681 raft_consensus.cc:3037] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:36.082613 5681 raft_consensus.cc:504] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:36.083153 5681 leader_election.cc:288] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 election: Requested vote from peers bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877), 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023)
I20260620 07:11:36.083559 5324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2731d3bb2a34f218156d8eabbad5e2d" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb38349d5a72479cab496909bc00ae43"
I20260620 07:11:36.083719 5324 raft_consensus.cc:3037] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:36.083710 5604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2731d3bb2a34f218156d8eabbad5e2d" candidate_uuid: "4808a6921503491ba8c0c7636997ebe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b39c7a16d54c198d1c97affb128ffc"
I20260620 07:11:36.083832 5604 raft_consensus.cc:3037] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:36.085458 5324 raft_consensus.cc:2454] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 1.
I20260620 07:11:36.085494 5604 raft_consensus.cc:2454] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4808a6921503491ba8c0c7636997ebe9 in term 1.
I20260620 07:11:36.085837 5398 leader_election.cc:302] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4808a6921503491ba8c0c7636997ebe9, bb38349d5a72479cab496909bc00ae43; no voters:
I20260620 07:11:36.086138 5681 raft_consensus.cc:2781] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:36.086277 5681 raft_consensus.cc:686] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [term 1 LEADER]: Becoming Leader. State: Replica: 4808a6921503491ba8c0c7636997ebe9, State: Running, Role: LEADER
I20260620 07:11:36.086405 5681 consensus_queue.cc:237] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } }
I20260620 07:11:36.087181 5705 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: TxnStatusTablet state changed. Reason: New leader 4808a6921503491ba8c0c7636997ebe9. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.087296 5705 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:36.087612 5717 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:11:36.087790 5717 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:11:36.088469 5162 catalog_manager.cc:5495] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2). New cstate: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:36.121071 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.116030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37759 (local address 127.0.0.1:40936)
0620 07:11:36.116302 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:36.116316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:36.116598 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:36.117034 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:36.117038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:36.117049 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:36.117186 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:36.117192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.118140 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.118145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:36.119031 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:36.119042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.119570 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.119574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:36.119670 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.119726 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:36.120702 (+ 976us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:36.120721 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:11:36.120791 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:11:36.120930 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 07:11:36.121582 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.116087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.3.245.2:37759)
0620 07:11:36.116224 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:36.116229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:36.116489 (+ 260us) server_negotiation.cc:407] Connection header received
0620 07:11:36.116820 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:36.116824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:36.116899 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:36.117009 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:36.117294 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.118023 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.119153 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.119447 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.119566 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.119882 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:36.120580 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:36.120739 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:36.121402 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 07:11:36.121486 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:11:36.124842 5604 raft_consensus.cc:1261] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Refusing update from remote peer 4808a6921503491ba8c0c7636997ebe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:36.125008 5324 raft_consensus.cc:1261] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Refusing update from remote peer 4808a6921503491ba8c0c7636997ebe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:36.125417 5681 consensus_queue.cc:1035] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:36.126116 5681 consensus_queue.cc:1035] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:36.127604 5680 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: TxnStatusTablet state changed. Reason: New leader 4808a6921503491ba8c0c7636997ebe9. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.127740 5680 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:36.127952 5692 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: TxnStatusTablet state changed. Reason: New leader 4808a6921503491ba8c0c7636997ebe9. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.128062 5692 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:36.130980 5705 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.131107 5705 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:36.131495 5692 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.131613 5692 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P bb38349d5a72479cab496909bc00ae43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:36.131865 5687 mvcc.cc:205] Tried to move back new op lower bound from 7298824176121196544 to 7298824175971479552. Current Snapshot: MvccSnapshot[applied={T|T < 7298824176121196544}]
I20260620 07:11:36.141389 5681 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.141597 5681 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 4808a6921503491ba8c0c7636997ebe9: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:36.142661 5680 tablet_replica.cc:439] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4808a6921503491ba8c0c7636997ebe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 } } peers { permanent_uuid: "4808a6921503491ba8c0c7636997ebe9" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37759 } } peers { permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 } } }
I20260620 07:11:36.142813 5680 tablet_replica.cc:442] T e2731d3bb2a34f218156d8eabbad5e2d P 79b39c7a16d54c198d1c97affb128ffc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:36.146204 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.140108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.3.245.2:37759)
0620 07:11:36.140238 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:36.140243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:36.141244 (+ 1001us) server_negotiation.cc:407] Connection header received
0620 07:11:36.141301 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:36.141304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:36.141366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:36.141441 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:36.141832 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.142936 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.144819 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.145179 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.145286 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.145662 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:36.145754 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:36.145834 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:36.145997 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:11:36.146049 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:11:36.165560 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.159639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:37813 (local address 127.3.245.2:37759)
0620 07:11:36.159776 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:36.159781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:36.160900 (+ 1119us) server_negotiation.cc:407] Connection header received
0620 07:11:36.161075 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:36.161079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:36.161153 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:36.161250 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:36.161656 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.162826 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.163908 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.164364 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.164595 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.164915 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:36.165050 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:36.165142 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:36.165267 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 07:11:36.165391 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":50}
I20260620 07:11:36.165570 5724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.159599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37759 (local address 127.3.245.2:37813)
0620 07:11:36.160550 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:36.160572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:36.160969 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:36.161407 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:36.161410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:36.161427 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:36.161546 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:36.161551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.163081 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.163084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:36.163762 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:36.163772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.164600 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.164603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:36.164723 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.164772 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:36.165167 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:36.165172 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:11:36.165257 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:11:36.165419 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":32,"thread_start_us":638,"threads_started":1}
I20260620 07:11:36.175287 5604 raft_consensus.cc:1261] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Refusing update from remote peer 4808a6921503491ba8c0c7636997ebe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:36.175328 5324 raft_consensus.cc:1261] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Refusing update from remote peer 4808a6921503491ba8c0c7636997ebe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:36.175724 5705 consensus_queue.cc:1035] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b39c7a16d54c198d1c97affb128ffc" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:36.175895 5681 consensus_queue.cc:1035] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb38349d5a72479cab496909bc00ae43" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:36.233733 5167 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:36.233999 5167 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:36.237578 5167 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 643464ffbaff42fa88e12839fe2c4fba: Sending DeleteTablet for 3 replicas of tablet 34b213835ccf40b2a64a141a8f7842bf
I20260620 07:11:36.238341 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b213835ccf40b2a64a141a8f7842bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:36 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:40934
I20260620 07:11:36.238504 5304 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b213835ccf40b2a64a141a8f7842bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:36 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:48002
I20260620 07:11:36.238561 5699 tablet_replica.cc:330] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: stopping tablet replica
I20260620 07:11:36.238583 5584 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b213835ccf40b2a64a141a8f7842bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:36 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60108
I20260620 07:11:36.238692 5699 raft_consensus.cc:2229] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:36.238709 5697 tablet_replica.cc:330] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: stopping tablet replica
I20260620 07:11:36.238742 5698 tablet_replica.cc:330] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: stopping tablet replica
I20260620 07:11:36.238828 5697 raft_consensus.cc:2229] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:36.238834 5698 raft_consensus.cc:2229] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:36.238950 5697 raft_consensus.cc:2258] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:36.238958 5698 raft_consensus.cc:2258] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:36.239012 5699 raft_consensus.cc:2258] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:36.239990 5699 ts_tablet_manager.cc:1905] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:36.239989 5697 ts_tablet_manager.cc:1905] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:36.242244 5698 ts_tablet_manager.cc:1905] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:36.244520 5697 ts_tablet_manager.cc:1918] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:36.244657 5697 log.cc:1196] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/34b213835ccf40b2a64a141a8f7842bf
I20260620 07:11:36.244696 5699 ts_tablet_manager.cc:1918] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:36.244856 5699 log.cc:1196] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/34b213835ccf40b2a64a141a8f7842bf
I20260620 07:11:36.244959 5697 ts_tablet_manager.cc:1939] T 34b213835ccf40b2a64a141a8f7842bf P bb38349d5a72479cab496909bc00ae43: Deleting consensus metadata
I20260620 07:11:36.245131 5699 ts_tablet_manager.cc:1939] T 34b213835ccf40b2a64a141a8f7842bf P 4808a6921503491ba8c0c7636997ebe9: Deleting consensus metadata
I20260620 07:11:36.245823 5151 catalog_manager.cc:4841] TS 4808a6921503491ba8c0c7636997ebe9 (127.3.245.2:37759): tablet 34b213835ccf40b2a64a141a8f7842bf (table test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]) successfully deleted
I20260620 07:11:36.246081 5154 catalog_manager.cc:4841] TS bb38349d5a72479cab496909bc00ae43 (127.3.245.1:36877): tablet 34b213835ccf40b2a64a141a8f7842bf (table test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]) successfully deleted
I20260620 07:11:36.247907 5698 ts_tablet_manager.cc:1918] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:36.248001 5698 log.cc:1196] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/34b213835ccf40b2a64a141a8f7842bf
I20260620 07:11:36.248282 5698 ts_tablet_manager.cc:1939] T 34b213835ccf40b2a64a141a8f7842bf P 79b39c7a16d54c198d1c97affb128ffc: Deleting consensus metadata
I20260620 07:11:36.249152 5152 catalog_manager.cc:4841] TS 79b39c7a16d54c198d1c97affb128ffc (127.3.245.3:42023): tablet 34b213835ccf40b2a64a141a8f7842bf (table test-table [id=f8e9b90552ab4ce2a4b371bf524fe5d3]) successfully deleted
Jun 20 07:11:36 dist-test-slave-ldfr krb5kdc[5122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:11:36.259105 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.241029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.3.245.1:36877)
0620 07:11:36.241184 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:36.241190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:36.244515 (+ 3325us) server_negotiation.cc:407] Connection header received
0620 07:11:36.245151 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:36.245155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:36.245212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:36.245267 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:36.246743 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.247586 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.248936 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.249425 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.249543 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.253762 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:36.253777 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:36.253803 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:36.253839 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:36.256388 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:36.256978 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:36.256987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:36.256989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:36.257093 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:36.257666 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:36.257671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:36.257673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:36.258153 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:36.258436 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:36.258743 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:11:36.258923 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:11:36.259662 5364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48018
I20260620 07:11:36.271219 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:36.261880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.3.245.62:36563)
0620 07:11:36.262136 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:36.262141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:36.262158 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:11:36.262206 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:36.262209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:36.262261 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:36.262472 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:36.263123 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.263944 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.264877 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:36.265232 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:36.265357 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:36.267437 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:36.267442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:36.267445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:36.267491 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:36.269407 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:36.269928 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:36.269938 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:36.269940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:36.270027 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:36.270313 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:36.270318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:36.270319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:36.270523 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:36.270668 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:36.270909 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 07:11:36.271044 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
W20260620 07:11:36.271750 5167 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41246
I20260620 07:11:36.273880 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5239
I20260620 07:11:36.286581 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5379
I20260620 07:11:36.294859 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5519
I20260620 07:11:36.301235 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5135
2026-06-20T07:11:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2955 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:36 dist-test-slave-ldfr krb5kdc[5734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:36 dist-test-slave-ldfr krb5kdc[5734](info): set up 2 sockets
Jun 20 07:11:36 dist-test-slave-ldfr krb5kdc[5734](info): commencing operation
krb5kdc: starting...
W20260620 07:11:37.376559 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:37 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:37.504041 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34551
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:37.620754 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:37.621083 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:37.621170 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:37.625332 5747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:37.625415 5747 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:11:37.625445 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:37.625490 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:37.625525 5747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:37.625561 5747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:37.631031 5747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34551
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:37.632203 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:37.633332 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:11:37.643286 5747 server_base.cc:1034] running on GCE node
W20260620 07:11:37.643217 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:37.643229 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:37.643201 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:37.644176 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:37.645421 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:37.646625 5747 hybrid_clock.cc:648] HybridClock initialized: now 1781939497646591 us; error 49 us; skew 500 ppm
Jun 20 07:11:37 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:37.649574 5747 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:37.650488 5747 webserver.cc:460] Webserver started at http://127.3.245.62:41601/ using document root <none> and password file <none>
I20260620 07:11:37.651074 5747 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:37.651134 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:37.651302 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:37.653383 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "cde307b863b84711907aca22cd35726a"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "74a8e75aaef051ae9f815f9c650c5aab"
server_key_iv: "486e14910ea0fdb43d27179c76c5e07d"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.653967 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "cde307b863b84711907aca22cd35726a"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "74a8e75aaef051ae9f815f9c650c5aab"
server_key_iv: "486e14910ea0fdb43d27179c76c5e07d"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.657769 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:37.660283 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:37.661743 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:37.661875 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "cde307b863b84711907aca22cd35726a"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "74a8e75aaef051ae9f815f9c650c5aab"
server_key_iv: "486e14910ea0fdb43d27179c76c5e07d"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.661983 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:37.672194 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:37.675554 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:37.675717 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:37.684059 5747 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:34551
I20260620 07:11:37.684067 5824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:34551 every 8 connection(s)
I20260620 07:11:37.685081 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:37.688002 5825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:37.691449 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5747
I20260620 07:11:37.691545 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:37.691877 4052 external_mini_cluster.cc:1428] Setting key 5e82cd7084da7b84b5ab75b64f267081
I20260620 07:11:37.694201 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Bootstrap starting.
I20260620 07:11:37.696962 5825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:37.697871 5825 log.cc:824] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:37.700260 5825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: No bootstrap required, opened a new log
Jun 20 07:11:37 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:37.705003 5825 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } }
I20260620 07:11:37.705295 5825 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:37.705355 5825 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde307b863b84711907aca22cd35726a, State: Initialized, Role: FOLLOWER
I20260620 07:11:37.705888 5825 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } }
I20260620 07:11:37.706049 5825 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:37.706120 5825 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:37.706204 5825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:37.707382 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:37.693273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.3.245.62:34551)
0620 07:11:37.693684 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:37.693700 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:37.693737 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:11:37.694432 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:37.694455 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:37.694791 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:37.695070 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:37.696029 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:37.697186 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:37.698369 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:37.698971 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:37.699105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:37.701988 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:37.701995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:37.702013 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:37.702044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:37.704088 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:37.704559 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:37.704565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:37.704570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:37.704735 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:37.705005 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:37.705008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:37.705010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:37.705527 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:37.705723 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:37.706052 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 07:11:37.706369 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260620 07:11:37.707353 5825 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } }
I20260620 07:11:37.707769 5825 leader_election.cc:302] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde307b863b84711907aca22cd35726a; no voters:
I20260620 07:11:37.708132 5825 leader_election.cc:288] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:37.708292 5830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:37.708527 5830 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [term 1 LEADER]: Becoming Leader. State: Replica: cde307b863b84711907aca22cd35726a, State: Running, Role: LEADER
I20260620 07:11:37.708963 5830 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } }
I20260620 07:11:37.709460 5825 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:37.710618 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde307b863b84711907aca22cd35726a. Latest consensus state: current_term: 1 leader_uuid: "cde307b863b84711907aca22cd35726a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } } }
I20260620 07:11:37.710748 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:37.711099 5831 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde307b863b84711907aca22cd35726a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde307b863b84711907aca22cd35726a" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34551 } } }
I20260620 07:11:37.711203 5831 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:37.711287 5839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:37.714742 5839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:37.720711 5839 catalog_manager.cc:1347] Generated new cluster ID: 6899b96964734a56b9a8e063c19a4285
I20260620 07:11:37.720803 5839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:37.741401 5839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:37.742405 5839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:37.754858 5839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Generated new TSK 0
I20260620 07:11:37.755576 5839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:37.789983 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34551
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:37.904166 5851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:37.904507 5851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:37.904592 5851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:37.908820 5851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:37.908931 5851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:37.909005 5851 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:11:37.909049 5851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:37.909121 5851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:37.914868 5851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:37.915987 5851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:37.917062 5851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:37.927032 5856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:37.927098 5857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:37.927031 5859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:37.927706 5851 server_base.cc:1034] running on GCE node
I20260620 07:11:37.928159 5851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:37.928828 5851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:37.929992 5851 hybrid_clock.cc:648] HybridClock initialized: now 1781939497929976 us; error 32 us; skew 500 ppm
Jun 20 07:11:37 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:37.937616 5851 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:37.939080 5851 webserver.cc:460] Webserver started at http://127.3.245.1:34511/ using document root <none> and password file <none>
I20260620 07:11:37.939877 5851 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:37.939988 5851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:37.940251 5851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:37.943126 5851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "965f675d2b364d0b92a075833df35055"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "11253e0af9a4406a1986f791bd3e3021"
server_key_iv: "04ddc8c209e0a76cc5139d59ee79f3a1"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.943815 5851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "965f675d2b364d0b92a075833df35055"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "11253e0af9a4406a1986f791bd3e3021"
server_key_iv: "04ddc8c209e0a76cc5139d59ee79f3a1"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.948776 5851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:11:37.952023 5866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:37.953408 5851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:37.953545 5851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "965f675d2b364d0b92a075833df35055"
format_stamp: "Formatted at 2026-06-20 07:11:37 on dist-test-slave-ldfr"
server_key: "11253e0af9a4406a1986f791bd3e3021"
server_key_iv: "04ddc8c209e0a76cc5139d59ee79f3a1"
server_key_version: "encryptionkey@0"
I20260620 07:11:37.953673 5851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:37.992115 5851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:37.995278 5851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:37.995486 5851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:37.997114 5851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:37.997198 5851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:37.997259 5851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:37.997319 5851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:38.008289 5851 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37819
I20260620 07:11:38.008322 5981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37819 every 8 connection(s)
I20260620 07:11:38.009337 5851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:38.011863 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:37.998229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:60623 (local address 127.3.245.62:34551)
0620 07:11:37.998390 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:37.998395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:37.999288 (+ 893us) server_negotiation.cc:407] Connection header received
0620 07:11:38.000630 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.000638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.000711 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.000818 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.002012 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.002951 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.004431 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.004828 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.004956 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.007721 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.007724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.007726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.007796 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.009549 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.010198 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.010203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.010204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.010303 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.010697 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.010700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.010702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.010902 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.011006 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.011550 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.011692 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:11:38.012665 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:37.998663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.1:60623)
0620 07:11:37.999123 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:37.999173 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.000339 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.001025 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.001032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.001380 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.001870 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.001884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.003100 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.003106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.004282 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.004292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.005010 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.005014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.005220 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.005855 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.005872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.007519 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.009663 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.009672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.009690 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.010047 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.010433 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.010437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.010442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.010588 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.011017 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.011025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.011313 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.011495 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.011810 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
I20260620 07:11:38.013969 5982 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34551
I20260620 07:11:38.014122 5982 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:38.014624 5982 heartbeater.cc:511] Master 127.3.245.62:34551 requested a full tablet report, sending...
I20260620 07:11:38.016330 5778 ts_manager.cc:194] Registered new tserver with Master: 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819)
I20260620 07:11:38.017774 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5851
I20260620 07:11:38.017903 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:38.017855 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:60623
I20260620 07:11:38.018179 4052 external_mini_cluster.cc:1428] Setting key 3b0f1420d38e6a4033acddbb97141a0b
I20260620 07:11:38.023747 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.016177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.1:34933)
0620 07:11:38.016478 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.016494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.016594 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.016861 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.016864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.017076 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.017198 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.017205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.017983 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.017986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.018766 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.018776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.019231 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.019234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.019328 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.019861 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.019885 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.020352 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.022202 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.022208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.022211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.022498 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.022815 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.022817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.022819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.022895 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.023322 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.023324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.023414 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.023494 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.023597 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":91,"threads_started":1}
I20260620 07:11:38.023927 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.016297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:34933 (local address 127.3.245.62:34551)
0620 07:11:38.016413 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.016434 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.016552 (+ 118us) server_negotiation.cc:407] Connection header received
0620 07:11:38.016704 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.016707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.016757 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.016820 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.017308 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.017805 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.018901 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.019118 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.019229 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.020504 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.020508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.020511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.020538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.022085 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.022610 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.022617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.022621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.022712 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.022982 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.022988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.022993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.023232 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.023335 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.023582 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.023720 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:38.048177 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34551
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:38.159744 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:38.160058 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:38.160140 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:38.164420 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:38.164515 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:38.164585 5991 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:11:38.164649 5991 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:38.164714 5991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:38.170416 5991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:38.171665 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:38.172741 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:38.182510 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:38.182518 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:38.182533 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:38.182963 5991 server_base.cc:1034] running on GCE node
I20260620 07:11:38.183369 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:38.184091 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:38.185313 5991 hybrid_clock.cc:648] HybridClock initialized: now 1781939498185291 us; error 33 us; skew 500 ppm
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:38.188308 5991 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:38.189278 5991 webserver.cc:460] Webserver started at http://127.3.245.2:32937/ using document root <none> and password file <none>
I20260620 07:11:38.189961 5991 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:38.190088 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:38.190310 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:38.192390 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "b695340011e648d6802a4ed1d6bfef8d"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "12d6859a262c93faaefaacddc8de0da6"
server_key_iv: "077235a4726767d232c8e0e16c23a9b9"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.193022 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "b695340011e648d6802a4ed1d6bfef8d"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "12d6859a262c93faaefaacddc8de0da6"
server_key_iv: "077235a4726767d232c8e0e16c23a9b9"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.196885 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:38.199455 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:38.200758 5991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:11:38.200891 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "b695340011e648d6802a4ed1d6bfef8d"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "12d6859a262c93faaefaacddc8de0da6"
server_key_iv: "077235a4726767d232c8e0e16c23a9b9"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.201004 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:38.217865 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:38.221354 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:38.221570 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:38.223013 5991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:38.223088 5991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:38.223163 5991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:38.223215 5991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:38.234524 5991 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:33983
I20260620 07:11:38.234596 6121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:33983 every 8 connection(s)
I20260620 07:11:38.235692 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:38.237949 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.224191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:33927 (local address 127.3.245.62:34551)
0620 07:11:38.224336 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.224344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.225473 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 07:11:38.226685 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.226689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.226759 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.226870 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.228660 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.229315 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.230648 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.230885 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.231001 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.233809 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.233813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.233816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.233849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.235580 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.236292 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.236296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.236298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.236394 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.236806 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.236810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.236812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.237014 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.237124 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.237648 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.237777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:11:38.238729 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.224747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.2:33927)
0620 07:11:38.225283 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.225327 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.226423 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.227115 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.227128 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.227635 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.228480 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.228505 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.229444 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.229448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.230498 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.230508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.231062 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.231067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.231237 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.231870 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.231886 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.233633 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.235728 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.235734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.235746 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.236160 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.236484 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.236487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.236490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.236686 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.237122 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.237129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.237372 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.237563 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.237905 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260620 07:11:38.240007 6122 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34551
I20260620 07:11:38.240157 6122 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:38.240796 6122 heartbeater.cc:511] Master 127.3.245.62:34551 requested a full tablet report, sending...
I20260620 07:11:38.242009 5779 ts_manager.cc:194] Registered new tserver with Master: b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983)
I20260620 07:11:38.242914 5779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:33927
I20260620 07:11:38.245226 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 5991
I20260620 07:11:38.245343 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:38.245668 4052 external_mini_cluster.cc:1428] Setting key 38fcafb00c06b9d084d086f7e2f4278c
I20260620 07:11:38.251616 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.242733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:39703 (local address 127.3.245.62:34551)
0620 07:11:38.242860 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.242864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.243049 (+ 185us) server_negotiation.cc:407] Connection header received
0620 07:11:38.243166 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.243169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.243212 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.243319 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.243850 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.244402 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.245630 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.245906 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.245993 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.247477 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.247480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.247482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.247503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.249333 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.249828 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.249831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.249833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.249932 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.250326 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.250329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.250330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.250776 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.250878 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.251335 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.251462 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:11:38.251686 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.242618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.2:39703)
0620 07:11:38.242928 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.242941 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.243063 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.243363 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.243369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.243563 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.243719 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.243726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.244536 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.244540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.245503 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.245515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.246028 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.246033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.246146 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.246750 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.246763 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.247310 (+ 547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.249498 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.249501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.249503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.249714 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.250037 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.250040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.250041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.250182 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.250931 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.250937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.251061 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.251184 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.251387 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:38.276774 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34551
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:38.395210 6131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:38.395557 6131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:38.395640 6131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:38.399955 6131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:38.400065 6131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:38.400147 6131 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 07:11:38.400193 6131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:11:38.400261 6131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:38.406060 6131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:38.407205 6131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:38.408284 6131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:38.418562 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:38.418609 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:38.418849 6131 server_base.cc:1034] running on GCE node
W20260620 07:11:38.419644 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:38.420092 6131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:38.420971 6131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:38.422194 6131 hybrid_clock.cc:648] HybridClock initialized: now 1781939498422174 us; error 29 us; skew 500 ppm
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:38.425359 6131 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:38.426440 6131 webserver.cc:460] Webserver started at http://127.3.245.3:45949/ using document root <none> and password file <none>
I20260620 07:11:38.427152 6131 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:38.427256 6131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:38.427542 6131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:38.429952 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "a27b35b586c84071b98a02f36caf28a0"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "50801217ae940445f6796ce2baf610f0"
server_key_iv: "fafae3b1a8e13cb5d4996f1008289c63"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.430601 6131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "a27b35b586c84071b98a02f36caf28a0"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "50801217ae940445f6796ce2baf610f0"
server_key_iv: "fafae3b1a8e13cb5d4996f1008289c63"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.434892 6131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:11:38.437902 6146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:38.439324 6131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:11:38.439483 6131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "a27b35b586c84071b98a02f36caf28a0"
format_stamp: "Formatted at 2026-06-20 07:11:38 on dist-test-slave-ldfr"
server_key: "50801217ae940445f6796ce2baf610f0"
server_key_iv: "fafae3b1a8e13cb5d4996f1008289c63"
server_key_version: "encryptionkey@0"
I20260620 07:11:38.439615 6131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:38.459596 6131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:38.463181 6131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:38.463411 6131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:38.465031 6131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:38.465108 6131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:38.465183 6131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:38.465226 6131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:38.476080 6131 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:38065
I20260620 07:11:38.476156 6261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:38065 every 8 connection(s)
I20260620 07:11:38.477289 6131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:38.479801 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.465896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:57859 (local address 127.3.245.62:34551)
0620 07:11:38.466053 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.466058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.467071 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 07:11:38.468162 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.468168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.468242 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.468350 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.469686 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.470591 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.471765 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.472169 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.472292 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.475515 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.475518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.475520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.475547 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.477542 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.478306 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.478310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.478312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.478413 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.478752 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.478754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.478756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.479005 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.479089 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.479574 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.479680 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:11:38.480722 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.466400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.3:57859)
0620 07:11:38.466884 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.466935 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.467919 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.468563 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.468574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.469066 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.469550 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.469562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.470733 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.470738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.471657 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.471667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.472390 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.472395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.472655 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.473359 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.473383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.475285 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.477725 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.477733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.477760 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.478174 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.478523 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.478525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.478527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.478665 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.479123 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.479128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.479357 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.479530 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.479782 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260620 07:11:38.482025 6262 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34551
I20260620 07:11:38.482175 6262 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:38.482820 6262 heartbeater.cc:511] Master 127.3.245.62:34551 requested a full tablet report, sending...
I20260620 07:11:38.484107 5778 ts_manager.cc:194] Registered new tserver with Master: a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:38.484201 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6131
I20260620 07:11:38.484301 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:38.484601 4052 external_mini_cluster.cc:1428] Setting key 7aaa383d84be2e6fdc5346c890dc3ada
I20260620 07:11:38.485069 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:57859
I20260620 07:11:38.486855 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:38.493489 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.484489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:45389 (local address 127.3.245.62:34551)
0620 07:11:38.484633 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.484638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.484872 (+ 234us) server_negotiation.cc:407] Connection header received
0620 07:11:38.484989 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.484992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.485038 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.485129 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.485699 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.486326 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.487592 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.487910 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.488017 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.489492 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.489497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.489502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.489536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.491469 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.492025 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.492029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.492031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.492125 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.492413 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.492418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.492422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.492701 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.492796 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.493231 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.493330 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:11:38.493742 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.484442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.3.245.3:45389)
0620 07:11:38.484781 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.484793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.484910 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.485175 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.485179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.485417 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.485545 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.485552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.486486 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.486489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.487457 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.487468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.488046 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.488053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.488184 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.488959 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.488979 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.489343 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.491603 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.491609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.491611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.491915 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.492222 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.492224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.492225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.492307 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.492910 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.492914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.493032 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.493211 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.493400 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:38.515610 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.506403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.3.245.62:34551)
0620 07:11:38.506531 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.506534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.506547 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:11:38.506588 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.506590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.506629 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.506699 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.507470 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.508005 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.509002 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.509322 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.509423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.512124 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.512128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.512130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.512158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.513845 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.514333 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.514337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.514338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.514422 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.514704 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.514707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.514709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.514928 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.515071 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.515312 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.515444 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:11:38.518632 5779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:38.521776 5779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:38.536995 6279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.530820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:38065 (local address 127.0.0.1:49668)
0620 07:11:38.531187 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.531240 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.531377 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.532275 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.532281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.532319 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.532496 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.532501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534140 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.534143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.535392 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.535402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536439 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.536572 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.536638 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.536753 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.536868 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":22,"spinlock_wait_cycles":2432,"thread_start_us":103,"threads_started":1}
I20260620 07:11:38.537046 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.530980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.3.245.3:38065)
0620 07:11:38.531692 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.531704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.531787 (+ 83us) server_negotiation.cc:407] Connection header received
0620 07:11:38.531870 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.531875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.532095 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.532685 (+ 590us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.532839 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534005 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.535515 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536280 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536543 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.536737 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.536791 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.536893 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":110,"threads_started":1}
I20260620 07:11:38.537324 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.531464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33983 (local address 127.0.0.1:45884)
0620 07:11:38.532203 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.532218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.532326 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.532899 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.532903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.532920 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.533008 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.533013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534345 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.534348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.535650 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.535661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536890 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.536989 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.537012 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.537097 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.537169 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":76,"spinlock_wait_cycles":13824,"thread_start_us":116,"threads_started":1}
I20260620 07:11:38.537422 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.531778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.3.245.2:33983)
0620 07:11:38.532228 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.532245 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.532273 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:11:38.532477 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.532482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.532709 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.532883 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.533143 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534215 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.535777 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536561 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.536820 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.537150 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.537256 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":103,"threads_started":1}
I20260620 07:11:38.538645 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.531538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.0.0.1:55218)
0620 07:11:38.532290 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.532304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.532372 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.533085 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.533088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.533106 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.533220 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.533226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534175 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.534178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.535693 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.535704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536465 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.538338 (+ 1870us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.538364 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.538449 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.538518 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":172,"thread_start_us":89,"threads_started":1}
I20260620 07:11:38.538726 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.531924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.3.245.1:37819)
0620 07:11:38.532588 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.532594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.532657 (+ 63us) server_negotiation.cc:407] Connection header received
0620 07:11:38.532727 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.532731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.532960 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.533104 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.533337 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.534045 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.535833 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.536337 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.536465 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.536625 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.538498 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.538602 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":57,"threads_started":1}
I20260620 07:11:38.540200 6056 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 (DEFAULT_TABLE table=test-table [id=15d961ced9ed401e98b7203631c9c501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.540200 6196 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 (DEFAULT_TABLE table=test-table [id=15d961ced9ed401e98b7203631c9c501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.541184 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51fd506ff400abb2016b6a3b37a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.541184 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51fd506ff400abb2016b6a3b37a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.546419 5916 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 (DEFAULT_TABLE table=test-table [id=15d961ced9ed401e98b7203631c9c501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.547550 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea51fd506ff400abb2016b6a3b37a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.548202 6285 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Bootstrap starting.
I20260620 07:11:38.549419 6286 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Bootstrap starting.
I20260620 07:11:38.550422 6285 tablet_bootstrap.cc:654] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.551163 6285 log.cc:824] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:38.551522 6286 tablet_bootstrap.cc:654] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.552356 6286 log.cc:824] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:38.553293 6285 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: No bootstrap required, opened a new log
I20260620 07:11:38.553503 6285 ts_tablet_manager.cc:1397] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 07:11:38.554598 6286 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: No bootstrap required, opened a new log
I20260620 07:11:38.554803 6286 ts_tablet_manager.cc:1397] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:38.554847 6289 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Bootstrap starting.
I20260620 07:11:38.556859 6289 tablet_bootstrap.cc:654] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.557641 6289 log.cc:824] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:38.557808 6285 raft_consensus.cc:348] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.558082 6285 raft_consensus.cc:374] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.558140 6285 raft_consensus.cc:729] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.558636 6285 consensus_queue.cc:260] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.558642 6286 raft_consensus.cc:348] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.558887 6286 raft_consensus.cc:374] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.558959 6286 raft_consensus.cc:729] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.559396 6286 consensus_queue.cc:260] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.559705 6285 ts_tablet_manager.cc:1428] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 07:11:38.559811 6289 tablet_bootstrap.cc:492] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: No bootstrap required, opened a new log
I20260620 07:11:38.560042 6122 heartbeater.cc:503] Master 127.3.245.62:34551 was elected leader, sending a full tablet report...
I20260620 07:11:38.560057 6289 ts_tablet_manager.cc:1397] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:11:38.560330 6286 ts_tablet_manager.cc:1428] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 07:11:38.560683 6262 heartbeater.cc:503] Master 127.3.245.62:34551 was elected leader, sending a full tablet report...
I20260620 07:11:38.564742 6289 raft_consensus.cc:348] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.565058 6289 raft_consensus.cc:374] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.565119 6289 raft_consensus.cc:729] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f675d2b364d0b92a075833df35055, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.565582 6289 consensus_queue.cc:260] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.566263 5982 heartbeater.cc:503] Master 127.3.245.62:34551 was elected leader, sending a full tablet report...
I20260620 07:11:38.566627 6289 ts_tablet_manager.cc:1428] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:11:38.589882 6291 raft_consensus.cc:482] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:38.590137 6291 raft_consensus.cc:504] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.591467 6291 leader_election.cc:288] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065), 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819)
I20260620 07:11:38.596289 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.591933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:38065 (local address 127.3.245.2:49709)
0620 07:11:38.592068 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.592104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.592211 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.592570 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.592574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.592603 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.592823 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.592837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.594018 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.594021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.595251 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.595261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.595969 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.595972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.596048 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.596068 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.596127 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.596183 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:11:38.596359 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.592041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:49709 (local address 127.3.245.3:38065)
0620 07:11:38.592186 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.592191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.592207 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:11:38.592341 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.592344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.592404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.592498 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.592984 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.593894 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.595394 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.595816 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.595985 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.596050 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.596165 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.596232 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:11:38.597093 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.592408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:60875 (local address 127.3.245.1:37819)
0620 07:11:38.592675 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.592680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.592746 (+ 66us) server_negotiation.cc:407] Connection header received
0620 07:11:38.592983 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.592986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.593067 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.593147 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.593574 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.594248 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.595853 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.596537 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.596675 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.596738 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.596941 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.597004 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:11:38.597105 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.592268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.3.245.2:60875)
0620 07:11:38.592647 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.592661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.592820 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.593180 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.593184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.593245 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.593436 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.593445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.594388 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.594392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.595731 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.595739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.596719 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.596723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.596791 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.596840 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.596916 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.596979 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260620 07:11:38.597141 6216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51fd506ff400abb2016b6a3b37a81" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0" is_pre_election: true
I20260620 07:11:38.597429 6216 raft_consensus.cc:2454] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 0.
I20260620 07:11:38.597995 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51fd506ff400abb2016b6a3b37a81" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055" is_pre_election: true
I20260620 07:11:38.598034 6007 leader_election.cc:302] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a27b35b586c84071b98a02f36caf28a0, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:38.598335 5936 raft_consensus.cc:2454] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 0.
I20260620 07:11:38.598333 6291 raft_consensus.cc:2781] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:38.598424 6291 raft_consensus.cc:482] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:38.598467 6291 raft_consensus.cc:3037] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.599601 6291 raft_consensus.cc:504] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.599944 6291 leader_election.cc:288] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 election: Requested vote from peers a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065), 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819)
I20260620 07:11:38.600281 6216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51fd506ff400abb2016b6a3b37a81" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0"
I20260620 07:11:38.600354 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea51fd506ff400abb2016b6a3b37a81" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055"
I20260620 07:11:38.600409 6216 raft_consensus.cc:3037] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.600517 5936 raft_consensus.cc:3037] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.601536 6216 raft_consensus.cc:2454] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 1.
I20260620 07:11:38.601691 5936 raft_consensus.cc:2454] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 1.
I20260620 07:11:38.601884 6007 leader_election.cc:302] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a27b35b586c84071b98a02f36caf28a0, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:38.602047 6291 raft_consensus.cc:2781] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:38.602283 6291 raft_consensus.cc:686] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 1 LEADER]: Becoming Leader. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Running, Role: LEADER
I20260620 07:11:38.602618 6291 consensus_queue.cc:237] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } }
I20260620 07:11:38.605844 5779 catalog_manager.cc:5495] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d reported cstate change: term changed from 0 to 1, leader changed from <none> to b695340011e648d6802a4ed1d6bfef8d (127.3.245.2). New cstate: current_term: 1 leader_uuid: "b695340011e648d6802a4ed1d6bfef8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } health_report { overall_health: HEALTHY } } }
Jun 20 07:11:38 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939497, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:38.647332 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.638565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34551 (local address 127.0.0.1:34254)
0620 07:11:38.638910 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.638923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.639028 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.639261 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.639264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.639637 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:38.639780 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.639826 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.640973 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.640976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.641600 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.641610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.642041 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.642043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.642104 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.642724 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:38.642736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:38.644254 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:38.645922 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.645930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.645936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.646163 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.646412 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:38.646414 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:38.646415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:38.646491 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:38.646818 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:38.646823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:38.646943 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.647036 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.647158 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":140,"threads_started":1}
I20260620 07:11:38.647339 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.638615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.3.245.62:34551)
0620 07:11:38.638766 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.638771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.639003 (+ 232us) server_negotiation.cc:407] Connection header received
0620 07:11:38.639110 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.639112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.639152 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.639219 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:38.639959 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.640668 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.641720 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.641938 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.642017 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.644388 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:38.644391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:38.644393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:38.644413 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:38.645816 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.646246 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.646248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.646249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.646318 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:38.646565 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:38.646567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:38.646568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:38.646730 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:38.646802 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.647092 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.647215 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":63}
I20260620 07:11:38.651382 5779 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:34254:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:11:38.652230 5779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:38.657408 6056 tablet_service.cc:1467] Processing CreateTablet for tablet a58b0d25b0bf4ee797c1a828b95917f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c71678d2f3764e5e9453a46055e61a06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:38.657370 5916 tablet_service.cc:1467] Processing CreateTablet for tablet a58b0d25b0bf4ee797c1a828b95917f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c71678d2f3764e5e9453a46055e61a06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:38.658169 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58b0d25b0bf4ee797c1a828b95917f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.658169 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58b0d25b0bf4ee797c1a828b95917f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.657873 6196 tablet_service.cc:1467] Processing CreateTablet for tablet a58b0d25b0bf4ee797c1a828b95917f4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c71678d2f3764e5e9453a46055e61a06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:11:38.658403 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58b0d25b0bf4ee797c1a828b95917f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.660652 6285 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: Bootstrap starting.
I20260620 07:11:38.661154 6286 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: Bootstrap starting.
I20260620 07:11:38.661695 6285 tablet_bootstrap.cc:654] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.662120 6286 tablet_bootstrap.cc:654] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.662760 6289 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: Bootstrap starting.
I20260620 07:11:38.663089 6285 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: No bootstrap required, opened a new log
I20260620 07:11:38.663182 6285 ts_tablet_manager.cc:1397] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:38.663744 6289 tablet_bootstrap.cc:654] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.663679 6285 raft_consensus.cc:348] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.663808 6285 raft_consensus.cc:374] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.663841 6285 raft_consensus.cc:729] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.664018 6285 consensus_queue.cc:260] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.664310 6291 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.664465 6291 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.664757 6285 ts_tablet_manager.cc:1428] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:38.664829 6286 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: No bootstrap required, opened a new log
I20260620 07:11:38.664935 6286 ts_tablet_manager.cc:1397] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 07:11:38.665423 6286 raft_consensus.cc:348] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.665549 6286 raft_consensus.cc:374] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.665580 6286 raft_consensus.cc:729] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.665701 6286 consensus_queue.cc:260] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.665822 6289 tablet_bootstrap.cc:492] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: No bootstrap required, opened a new log
I20260620 07:11:38.665916 6289 ts_tablet_manager.cc:1397] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:38.665974 6286 ts_tablet_manager.cc:1428] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:38.665941 6292 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.666067 6292 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.666481 6289 raft_consensus.cc:348] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.666603 6289 raft_consensus.cc:374] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.666643 6289 raft_consensus.cc:729] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f675d2b364d0b92a075833df35055, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.666778 6289 consensus_queue.cc:260] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.667076 6289 ts_tablet_manager.cc:1428] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:38.667088 6293 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.667251 6293 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 07:11:38.728410 6263 tablet.cc:2369] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:38.736850 6123 tablet.cc:2369] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:38.760921 5983 tablet.cc:2369] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:38.801898 6293 raft_consensus.cc:482] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:38.802055 6293 raft_consensus.cc:504] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.803350 6293 leader_election.cc:288] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:38.808349 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.803710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:38065 (local address 127.3.245.1:59419)
0620 07:11:38.804154 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.804169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.804339 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.804669 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.804675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.804698 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.804942 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.804950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.805833 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.805836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.807133 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.807144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.807958 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.807961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.808041 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.808063 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.808128 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.808186 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":29,"spinlock_wait_cycles":38656,"thread_start_us":156,"threads_started":1}
I20260620 07:11:38.808393 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.803879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:59419 (local address 127.3.245.3:38065)
0620 07:11:38.804007 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.804014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.804295 (+ 281us) server_negotiation.cc:407] Connection header received
0620 07:11:38.804485 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.804488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.804542 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.804606 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.805068 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.805719 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.807270 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.807804 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.807934 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.808002 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.808166 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.808243 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:11:38.808926 6216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58b0d25b0bf4ee797c1a828b95917f4" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0" is_pre_election: true
I20260620 07:11:38.809113 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.803849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:33589 (local address 127.3.245.2:33983)
0620 07:11:38.803992 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.803997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.804318 (+ 321us) server_negotiation.cc:407] Connection header received
0620 07:11:38.804511 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.804514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.804560 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.804703 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:38.805678 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.806259 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.807661 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.808367 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.808485 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.808552 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.808910 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.808985 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:11:38.809139 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.803710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33983 (local address 127.3.245.1:33589)
0620 07:11:38.804168 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.804181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.804340 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.804718 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.804722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.804773 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:38.805550 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.805560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.806377 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.806381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.807528 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.807545 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.808649 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.808653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.808758 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.808789 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.808875 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.808975 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":98,"spinlock_wait_cycles":1408,"thread_start_us":89,"threads_started":1}
I20260620 07:11:38.809149 6216 raft_consensus.cc:2454] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 0.
I20260620 07:11:38.809602 5867 leader_election.cc:302] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, a27b35b586c84071b98a02f36caf28a0; no voters:
I20260620 07:11:38.809782 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58b0d25b0bf4ee797c1a828b95917f4" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d" is_pre_election: true
I20260620 07:11:38.809937 6293 raft_consensus.cc:2781] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:38.810016 6293 raft_consensus.cc:482] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:38.810060 6076 raft_consensus.cc:2454] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 0.
I20260620 07:11:38.810106 6293 raft_consensus.cc:3037] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.811333 6293 raft_consensus.cc:504] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.811714 6293 leader_election.cc:288] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 election: Requested vote from peers b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:38.812137 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58b0d25b0bf4ee797c1a828b95917f4" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d"
I20260620 07:11:38.812270 6216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58b0d25b0bf4ee797c1a828b95917f4" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0"
I20260620 07:11:38.812318 6076 raft_consensus.cc:3037] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.812418 6216 raft_consensus.cc:3037] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:38.813452 6216 raft_consensus.cc:2454] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 1.
I20260620 07:11:38.813537 6076 raft_consensus.cc:2454] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 1.
I20260620 07:11:38.813772 5867 leader_election.cc:302] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, a27b35b586c84071b98a02f36caf28a0; no voters:
I20260620 07:11:38.813956 6293 raft_consensus.cc:2781] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:38.814188 6293 raft_consensus.cc:686] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [term 1 LEADER]: Becoming Leader. State: Replica: 965f675d2b364d0b92a075833df35055, State: Running, Role: LEADER
I20260620 07:11:38.814556 6293 consensus_queue.cc:237] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.817075 6308 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: TxnStatusTablet state changed. Reason: New leader 965f675d2b364d0b92a075833df35055. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.817205 6308 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:38.817546 6310 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:11:38.817807 6310 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:11:38.818280 5777 catalog_manager.cc:5495] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 965f675d2b364d0b92a075833df35055 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:38.864331 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.860249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.3.245.1:37819)
0620 07:11:38.860390 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.860393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.860425 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:11:38.860659 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.860664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.860732 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.860832 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:38.861147 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.861644 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.862871 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.863087 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.863175 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.863510 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.863935 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.864021 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.864124 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.864175 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:38.864354 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.860113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.0.0.1:55230)
0620 07:11:38.860368 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.860383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.860488 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.860862 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.860866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.860873 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:38.861001 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.861007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.861750 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.861753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.862718 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.862728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.863196 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.863200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.863305 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.863369 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.864025 (+ 656us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.864039 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.864114 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.864200 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 07:11:38.870342 6216 raft_consensus.cc:1261] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Refusing update from remote peer 965f675d2b364d0b92a075833df35055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:38.870383 6076 raft_consensus.cc:1261] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Refusing update from remote peer 965f675d2b364d0b92a075833df35055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:38.871058 6308 consensus_queue.cc:1035] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:38.871268 6293 consensus_queue.cc:1035] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:38.875111 6291 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: TxnStatusTablet state changed. Reason: New leader 965f675d2b364d0b92a075833df35055. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.875238 6291 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.875779 6292 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: TxnStatusTablet state changed. Reason: New leader 965f675d2b364d0b92a075833df35055. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.875896 6292 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.877862 6293 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.877981 6293 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:38.878086 6313 mvcc.cc:205] Tried to move back new op lower bound from 7298824187364196352 to 7298824187147563008. Current Snapshot: MvccSnapshot[applied={T|T < 7298824187364196352}]
I20260620 07:11:38.879253 6291 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.879387 6291 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P b695340011e648d6802a4ed1d6bfef8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.880506 6308 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.881742 6308 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P 965f675d2b364d0b92a075833df35055: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:11:38.882026 6315 mvcc.cc:205] Tried to move back new op lower bound from 7298824187364196352 to 7298824187147563008. Current Snapshot: MvccSnapshot[applied={T|T < 7298824187364196352}]
I20260620 07:11:38.882905 6292 tablet_replica.cc:439] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } } }
I20260620 07:11:38.883029 6292 tablet_replica.cc:442] T a58b0d25b0bf4ee797c1a828b95917f4 P a27b35b586c84071b98a02f36caf28a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:11:38.885445 6314 mvcc.cc:205] Tried to move back new op lower bound from 7298824187364196352 to 7298824187147563008. Current Snapshot: MvccSnapshot[applied={T|T < 7298824187364196352 or (T in {7298824187364196352})}]
I20260620 07:11:38.890138 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.886273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.3.245.2:33983)
0620 07:11:38.886380 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.886384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.886398 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:11:38.886551 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.886554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.886617 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.886852 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:38.887149 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.887692 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.888597 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.888872 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.888996 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.889305 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.889741 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.889868 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.889936 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.889993 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:11:38.902544 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.898528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:32823 (local address 127.3.245.1:37819)
0620 07:11:38.898666 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.898670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.898938 (+ 268us) server_negotiation.cc:407] Connection header received
0620 07:11:38.899173 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.899176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.899231 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.899315 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:38.899635 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.900143 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.901441 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.901653 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.901737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.902125 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.902202 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.902260 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.902377 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.902426 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:38.902608 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.898435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.3.245.2:32823)
0620 07:11:38.898818 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.898834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.898966 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.899339 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.899342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.899351 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:38.899488 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.899495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.900250 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.900254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.901300 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.901312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.901788 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.901792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.901896 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.901956 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.902283 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.902290 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.902355 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.902439 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":132,"threads_started":1}
I20260620 07:11:38.910480 6216 raft_consensus.cc:1261] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Refusing update from remote peer b695340011e648d6802a4ed1d6bfef8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:38.910753 5936 raft_consensus.cc:1261] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Refusing update from remote peer b695340011e648d6802a4ed1d6bfef8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:38.911113 6295 consensus_queue.cc:1035] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:38.911293 6291 consensus_queue.cc:1035] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:38.938159 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.933956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:57531 (local address 127.3.245.2:33983)
0620 07:11:38.934069 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:38.934073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:38.934088 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:11:38.934212 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:38.934215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:38.934278 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:38.934328 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:38.934729 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.935279 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.936874 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.937219 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.937366 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.937672 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.937760 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.937849 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:38.937957 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:11:38.938012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:11:38.938191 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:38.933594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33983 (local address 127.3.245.1:57531)
0620 07:11:38.933945 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:38.933963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:38.934070 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:38.934477 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:38.934482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:38.934491 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:38.934625 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.934632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.935424 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.935428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.936730 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:38.936751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:38.937372 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:38.937376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:38.937485 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:38.937531 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:38.937875 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:38.937881 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:11:38.937946 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:11:38.938034 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260620 07:11:38.966005 5777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:38.966202 5777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:38.969139 5777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Sending DeleteTablet for 3 replicas of tablet 2ea51fd506ff400abb2016b6a3b37a81
I20260620 07:11:38.970031 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:38 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 07:11:38.970638 5777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:11:38.971077 6332 tablet_replica.cc:330] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: stopping tablet replica
W20260620 07:11:38.971083 5777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:38.971484 6332 raft_consensus.cc:2229] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:38.971820 6332 raft_consensus.cc:2258] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:38.971863 5916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:38 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260620 07:11:38.972378 6333 tablet_replica.cc:330] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: stopping tablet replica
I20260620 07:11:38.972657 6056 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea51fd506ff400abb2016b6a3b37a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:38 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:45884
I20260620 07:11:38.972749 6333 raft_consensus.cc:2229] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:38.973002 6333 raft_consensus.cc:2258] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:38.974452 6333 ts_tablet_manager.cc:1905] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:38.976821 6334 tablet_replica.cc:330] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: stopping tablet replica
I20260620 07:11:38.977146 6334 raft_consensus.cc:2229] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:38.977567 6334 raft_consensus.cc:2258] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:38.977644 6056 tablet_service.cc:1467] Processing CreateTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 (DEFAULT_TABLE table=test-table [id=d355eac948df4452a344774a5da4f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.977701 6196 tablet_service.cc:1467] Processing CreateTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 (DEFAULT_TABLE table=test-table [id=d355eac948df4452a344774a5da4f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.977919 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1fb12c75114fc1b4578b0c062dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.977982 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1fb12c75114fc1b4578b0c062dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.980386 6285 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Bootstrap starting.
I20260620 07:11:38.980558 6286 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Bootstrap starting.
I20260620 07:11:38.981514 6285 tablet_bootstrap.cc:654] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.981514 6286 tablet_bootstrap.cc:654] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.982784 5916 tablet_service.cc:1467] Processing CreateTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 (DEFAULT_TABLE table=test-table [id=d355eac948df4452a344774a5da4f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:38.983033 6332 ts_tablet_manager.cc:1905] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:38.983062 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1fb12c75114fc1b4578b0c062dfb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:38.983417 6334 ts_tablet_manager.cc:1905] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:38.985821 6289 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Bootstrap starting.
I20260620 07:11:38.986436 6334 ts_tablet_manager.cc:1918] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:38.986526 6334 log.cc:1196] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/2ea51fd506ff400abb2016b6a3b37a81
I20260620 07:11:38.986856 6334 ts_tablet_manager.cc:1939] T 2ea51fd506ff400abb2016b6a3b37a81 P b695340011e648d6802a4ed1d6bfef8d: Deleting consensus metadata
I20260620 07:11:38.986925 6289 tablet_bootstrap.cc:654] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:38.987277 6332 ts_tablet_manager.cc:1918] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:38.987357 6332 log.cc:1196] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/2ea51fd506ff400abb2016b6a3b37a81
I20260620 07:11:38.987656 6332 ts_tablet_manager.cc:1939] T 2ea51fd506ff400abb2016b6a3b37a81 P a27b35b586c84071b98a02f36caf28a0: Deleting consensus metadata
I20260620 07:11:38.988821 6333 ts_tablet_manager.cc:1918] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:38.988974 6333 log.cc:1196] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/2ea51fd506ff400abb2016b6a3b37a81
I20260620 07:11:38.989354 6333 ts_tablet_manager.cc:1939] T 2ea51fd506ff400abb2016b6a3b37a81 P 965f675d2b364d0b92a075833df35055: Deleting consensus metadata
I20260620 07:11:38.989650 5763 catalog_manager.cc:4841] TS a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065): tablet 2ea51fd506ff400abb2016b6a3b37a81 (table test-table [id=15d961ced9ed401e98b7203631c9c501]) successfully deleted
I20260620 07:11:38.990602 5763 catalog_manager.cc:4841] TS 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819): tablet 2ea51fd506ff400abb2016b6a3b37a81 (table test-table [id=15d961ced9ed401e98b7203631c9c501]) successfully deleted
I20260620 07:11:38.991526 6289 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: No bootstrap required, opened a new log
I20260620 07:11:38.991631 6289 ts_tablet_manager.cc:1397] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 07:11:38.991755 6286 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: No bootstrap required, opened a new log
I20260620 07:11:38.991842 6286 ts_tablet_manager.cc:1397] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 07:11:38.992057 6289 raft_consensus.cc:348] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.992161 6289 raft_consensus.cc:374] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.992202 6289 raft_consensus.cc:729] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f675d2b364d0b92a075833df35055, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.992241 6286 raft_consensus.cc:348] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.992347 6286 raft_consensus.cc:374] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.992317 6289 consensus_queue.cc:260] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.992379 6286 raft_consensus.cc:729] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.992506 6286 consensus_queue.cc:260] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.992568 6289 ts_tablet_manager.cc:1428] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:38.992805 6286 ts_tablet_manager.cc:1428] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:11:38.997620 6285 tablet_bootstrap.cc:492] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: No bootstrap required, opened a new log
I20260620 07:11:38.997730 6285 ts_tablet_manager.cc:1397] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260620 07:11:38.997709 5765 catalog_manager.cc:4841] TS b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983): tablet 2ea51fd506ff400abb2016b6a3b37a81 (table test-table [id=15d961ced9ed401e98b7203631c9c501]) successfully deleted
I20260620 07:11:38.998162 6285 raft_consensus.cc:348] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.998306 6285 raft_consensus.cc:374] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:38.998356 6285 raft_consensus.cc:729] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Initialized, Role: FOLLOWER
I20260620 07:11:38.998500 6285 consensus_queue.cc:260] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:38.998790 6285 ts_tablet_manager.cc:1428] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:39.176887 6292 raft_consensus.cc:482] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:39.177042 6292 raft_consensus.cc:504] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:39.178532 6292 leader_election.cc:288] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819), b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983)
I20260620 07:11:39.184345 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.178786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.3.245.3:39597)
0620 07:11:39.179122 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:39.179139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:39.179249 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:39.180413 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:39.180416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:39.180430 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:39.180582 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.180589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.181756 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.181805 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.182921 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.182929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.183854 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.183858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.183974 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.183998 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:11:39.184098 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 07:11:39.184180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":88,"threads_started":1}
I20260620 07:11:39.184332 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.178935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:39597 (local address 127.3.245.1:37819)
0620 07:11:39.179741 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:39.179746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:39.180065 (+ 319us) server_negotiation.cc:407] Connection header received
0620 07:11:39.180140 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:39.180143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:39.180196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:39.180297 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:39.180794 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.181631 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.183107 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.183718 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.183850 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.183922 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:39.184117 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 07:11:39.184193 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658}
I20260620 07:11:39.184506 6283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.179195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:60897 (local address 127.3.245.2:33983)
0620 07:11:39.179420 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:39.179427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:39.179449 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:11:39.179518 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:39.179522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:39.179588 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:39.179705 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:39.180231 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.181137 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.182953 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.183818 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.183972 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.184051 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:39.184285 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 07:11:39.184366 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:11:39.184526 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.178871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33983 (local address 127.3.245.3:60897)
0620 07:11:39.179208 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:39.179220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:39.179299 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:39.179868 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:39.179875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:39.179906 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:39.180048 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.180060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.181331 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.181334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.182783 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.182794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.183974 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.183979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.184098 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.184126 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:11:39.184263 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:11:39.184355 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":76,"threads_started":1}
I20260620 07:11:39.185361 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1fb12c75114fc1b4578b0c062dfb40" candidate_uuid: "a27b35b586c84071b98a02f36caf28a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055" is_pre_election: true
I20260620 07:11:39.185423 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1fb12c75114fc1b4578b0c062dfb40" candidate_uuid: "a27b35b586c84071b98a02f36caf28a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d" is_pre_election: true
I20260620 07:11:39.185539 6076 raft_consensus.cc:2454] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27b35b586c84071b98a02f36caf28a0 in term 0.
I20260620 07:11:39.185539 5936 raft_consensus.cc:2454] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a27b35b586c84071b98a02f36caf28a0 in term 0.
I20260620 07:11:39.185922 6149 leader_election.cc:302] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a27b35b586c84071b98a02f36caf28a0, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:39.186249 6292 raft_consensus.cc:2781] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:39.186319 6292 raft_consensus.cc:482] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:39.186348 6292 raft_consensus.cc:3037] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:39.187359 6292 raft_consensus.cc:504] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:39.187788 6292 leader_election.cc:288] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [CANDIDATE]: Term 1 election: Requested vote from peers 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819), b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983)
I20260620 07:11:39.188124 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1fb12c75114fc1b4578b0c062dfb40" candidate_uuid: "a27b35b586c84071b98a02f36caf28a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055"
I20260620 07:11:39.188265 5936 raft_consensus.cc:3037] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:39.188315 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1fb12c75114fc1b4578b0c062dfb40" candidate_uuid: "a27b35b586c84071b98a02f36caf28a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d"
I20260620 07:11:39.188498 6076 raft_consensus.cc:3037] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:39.189370 5936 raft_consensus.cc:2454] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27b35b586c84071b98a02f36caf28a0 in term 1.
I20260620 07:11:39.189669 6076 raft_consensus.cc:2454] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a27b35b586c84071b98a02f36caf28a0 in term 1.
I20260620 07:11:39.189728 6147 leader_election.cc:302] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, a27b35b586c84071b98a02f36caf28a0; no voters:
I20260620 07:11:39.189939 6292 raft_consensus.cc:2781] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:39.190126 6292 raft_consensus.cc:686] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 1 LEADER]: Becoming Leader. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Running, Role: LEADER
I20260620 07:11:39.190394 6292 consensus_queue.cc:237] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:39.193104 5777 catalog_manager.cc:5495] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27b35b586c84071b98a02f36caf28a0 (127.3.245.3). New cstate: current_term: 1 leader_uuid: "a27b35b586c84071b98a02f36caf28a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } health_report { overall_health: HEALTHY } } }
I20260620 07:11:39.232925 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.229214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.3.245.3:38065)
0620 07:11:39.229336 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:39.229340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:39.229356 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:11:39.229470 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:39.229473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:39.229530 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:39.229633 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:39.229968 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.230538 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.231420 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.231632 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.231715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.231968 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:39.232424 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:39.232512 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:39.232599 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 07:11:39.232713 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:11:39.244517 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.240735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:36395 (local address 127.3.245.1:37819)
0620 07:11:39.240890 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:39.240893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:39.241023 (+ 130us) server_negotiation.cc:407] Connection header received
0620 07:11:39.241118 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:39.241121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:39.241246 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:39.241370 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:39.241655 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.242202 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.243455 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.243684 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.243763 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.244058 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:39.244125 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:39.244176 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:39.244323 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 07:11:39.244400 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:11:39.244542 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.240541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.3.245.3:36395)
0620 07:11:39.240893 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:39.240909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:39.241021 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:39.241382 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:39.241385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:39.241394 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:39.241520 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.241526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.242306 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.242309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.243301 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.243311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.243784 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.243788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.243882 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.243935 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:39.244198 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:39.244204 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:11:39.244264 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 07:11:39.244354 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":136,"threads_started":1}
I20260620 07:11:39.251466 6076 raft_consensus.cc:1261] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Refusing update from remote peer a27b35b586c84071b98a02f36caf28a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:39.251504 5936 raft_consensus.cc:1261] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Refusing update from remote peer a27b35b586c84071b98a02f36caf28a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:39.251904 6292 consensus_queue.cc:1035] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:39.252096 6340 consensus_queue.cc:1035] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:39.270539 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.266798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:38065 (local address 127.3.245.1:52899)
0620 07:11:39.266955 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:39.266970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:39.267039 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:39.267353 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:39.267356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:39.267364 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:39.267503 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.267510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.268335 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.268338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.269063 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:39.269073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.269733 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.269737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:39.269852 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.269892 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:39.270239 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:39.270249 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 07:11:39.270336 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:11:39.270429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 07:11:39.270529 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:39.266787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:52899 (local address 127.3.245.3:38065)
0620 07:11:39.266988 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:39.266993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:39.267010 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:11:39.267142 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:39.267154 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:39.267203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:39.267317 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:39.267616 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.268245 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.269194 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:39.269460 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:39.269561 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:39.270031 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:39.270125 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:39.270233 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:39.270349 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:11:39.270397 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260620 07:11:39.271363 6314 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:11:40.830361 5777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:40.830510 5777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:40.833034 5777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Sending DeleteTablet for 3 replicas of tablet bf1fb12c75114fc1b4578b0c062dfb40
I20260620 07:11:40.833593 5916 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260620 07:11:40.833632 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 07:11:40.833740 6056 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1fb12c75114fc1b4578b0c062dfb40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:45884
I20260620 07:11:40.833961 6370 tablet_replica.cc:330] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: stopping tablet replica
I20260620 07:11:40.834100 6370 raft_consensus.cc:2229] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:40.834229 6371 tablet_replica.cc:330] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: stopping tablet replica
I20260620 07:11:40.834241 6370 raft_consensus.cc:2258] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:40.834409 6371 raft_consensus.cc:2229] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:40.834786 6371 raft_consensus.cc:2258] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:40.834924 6370 ts_tablet_manager.cc:1905] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:40.835494 6371 ts_tablet_manager.cc:1905] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:40.837185 6372 tablet_replica.cc:330] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: stopping tablet replica
I20260620 07:11:40.837370 6372 raft_consensus.cc:2229] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:40.837522 6372 raft_consensus.cc:2258] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:40.837797 6370 ts_tablet_manager.cc:1918] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:11:40.837881 6370 log.cc:1196] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/bf1fb12c75114fc1b4578b0c062dfb40
I20260620 07:11:40.838151 6370 ts_tablet_manager.cc:1939] T bf1fb12c75114fc1b4578b0c062dfb40 P 965f675d2b364d0b92a075833df35055: Deleting consensus metadata
I20260620 07:11:40.838165 6372 ts_tablet_manager.cc:1905] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:40.838449 6371 ts_tablet_manager.cc:1918] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:11:40.838528 6371 log.cc:1196] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/bf1fb12c75114fc1b4578b0c062dfb40
I20260620 07:11:40.838821 6371 ts_tablet_manager.cc:1939] T bf1fb12c75114fc1b4578b0c062dfb40 P a27b35b586c84071b98a02f36caf28a0: Deleting consensus metadata
I20260620 07:11:40.839025 5763 catalog_manager.cc:4841] TS 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819): tablet bf1fb12c75114fc1b4578b0c062dfb40 (table test-table [id=d355eac948df4452a344774a5da4f13e]) successfully deleted
I20260620 07:11:40.839978 5763 catalog_manager.cc:4841] TS a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065): tablet bf1fb12c75114fc1b4578b0c062dfb40 (table test-table [id=d355eac948df4452a344774a5da4f13e]) successfully deleted
I20260620 07:11:40.840917 6372 ts_tablet_manager.cc:1918] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:11:40.841017 6372 log.cc:1196] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/bf1fb12c75114fc1b4578b0c062dfb40
I20260620 07:11:40.841266 6372 ts_tablet_manager.cc:1939] T bf1fb12c75114fc1b4578b0c062dfb40 P b695340011e648d6802a4ed1d6bfef8d: Deleting consensus metadata
I20260620 07:11:40.841954 5765 catalog_manager.cc:4841] TS b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983): tablet bf1fb12c75114fc1b4578b0c062dfb40 (table test-table [id=d355eac948df4452a344774a5da4f13e]) successfully deleted
Jun 20 07:11:40 dist-test-slave-ldfr krb5kdc[5734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:11:40 dist-test-slave-ldfr krb5kdc[5734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939500, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:40.869148 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:40.854646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.3.245.62:34551)
0620 07:11:40.854951 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:40.854958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:40.854974 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:11:40.855047 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:40.855049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:40.855091 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:40.855169 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:40.855977 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:40.856550 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:40.857574 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:40.857857 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:40.858013 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:40.865789 (+ 7776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:40.865793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:40.865795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:40.865825 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:40.867465 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:40.867871 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:40.867874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:40.867875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:40.867962 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:40.868243 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:40.868246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:40.868247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:40.868456 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:40.868586 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:40.868892 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:11:40.869013 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":108,"threads_started":1}
I20260620 07:11:40.872031 5777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:40.872510 5777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:40.877413 6196 tablet_service.cc:1467] Processing CreateTablet for tablet ad75eb79a10248aabf0523e21af41642 (DEFAULT_TABLE table=test-table [id=fb310972a4d74fe9a610131286dfff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:40.877413 5916 tablet_service.cc:1467] Processing CreateTablet for tablet ad75eb79a10248aabf0523e21af41642 (DEFAULT_TABLE table=test-table [id=fb310972a4d74fe9a610131286dfff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:40.877413 6056 tablet_service.cc:1467] Processing CreateTablet for tablet ad75eb79a10248aabf0523e21af41642 (DEFAULT_TABLE table=test-table [id=fb310972a4d74fe9a610131286dfff93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:40.877722 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75eb79a10248aabf0523e21af41642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:40.877722 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75eb79a10248aabf0523e21af41642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:40.877723 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad75eb79a10248aabf0523e21af41642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:40.880367 6381 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Bootstrap starting.
I20260620 07:11:40.880388 6383 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Bootstrap starting.
I20260620 07:11:40.880638 6382 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Bootstrap starting.
I20260620 07:11:40.881333 6383 tablet_bootstrap.cc:654] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:40.881418 6381 tablet_bootstrap.cc:654] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:40.881481 6382 tablet_bootstrap.cc:654] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:40.882582 6383 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: No bootstrap required, opened a new log
I20260620 07:11:40.882603 6381 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: No bootstrap required, opened a new log
I20260620 07:11:40.882680 6383 ts_tablet_manager.cc:1397] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:40.882714 6381 ts_tablet_manager.cc:1397] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:40.883123 6383 raft_consensus.cc:348] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.883150 6381 raft_consensus.cc:348] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.883266 6383 raft_consensus.cc:374] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:40.883311 6383 raft_consensus.cc:729] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Initialized, Role: FOLLOWER
I20260620 07:11:40.883445 6383 consensus_queue.cc:260] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.883757 6383 ts_tablet_manager.cc:1428] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:11:40.883790 6382 tablet_bootstrap.cc:492] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: No bootstrap required, opened a new log
I20260620 07:11:40.883875 6382 ts_tablet_manager.cc:1397] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:40.884279 6382 raft_consensus.cc:348] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.884406 6382 raft_consensus.cc:374] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:40.884440 6382 raft_consensus.cc:729] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f675d2b364d0b92a075833df35055, State: Initialized, Role: FOLLOWER
I20260620 07:11:40.884567 6382 consensus_queue.cc:260] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.883270 6381 raft_consensus.cc:374] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:40.884713 6381 raft_consensus.cc:729] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Initialized, Role: FOLLOWER
I20260620 07:11:40.884893 6382 ts_tablet_manager.cc:1428] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:40.884917 6381 consensus_queue.cc:260] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:40.885346 6381 ts_tablet_manager.cc:1428] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:41.050913 6387 raft_consensus.cc:482] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:41.051082 6387 raft_consensus.cc:504] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.051484 6387 leader_election.cc:288] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:41.052062 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75eb79a10248aabf0523e21af41642" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055" is_pre_election: true
I20260620 07:11:41.052225 5936 raft_consensus.cc:2454] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 0.
I20260620 07:11:41.052196 6215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75eb79a10248aabf0523e21af41642" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0" is_pre_election: true
I20260620 07:11:41.052402 6215 raft_consensus.cc:2454] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 0.
I20260620 07:11:41.052688 6007 leader_election.cc:302] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:41.053015 6387 raft_consensus.cc:2781] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:41.053076 6387 raft_consensus.cc:482] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:41.053107 6387 raft_consensus.cc:3037] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.054386 6387 raft_consensus.cc:504] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.054781 6387 leader_election.cc:288] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 election: Requested vote from peers 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:41.055302 5936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75eb79a10248aabf0523e21af41642" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965f675d2b364d0b92a075833df35055"
I20260620 07:11:41.055393 6215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad75eb79a10248aabf0523e21af41642" candidate_uuid: "b695340011e648d6802a4ed1d6bfef8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0"
I20260620 07:11:41.055464 5936 raft_consensus.cc:3037] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.055559 6215 raft_consensus.cc:3037] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.056391 5936 raft_consensus.cc:2454] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 1.
I20260620 07:11:41.056550 6215 raft_consensus.cc:2454] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b695340011e648d6802a4ed1d6bfef8d in term 1.
I20260620 07:11:41.056697 6007 leader_election.cc:302] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:41.056974 6387 raft_consensus.cc:2781] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:41.057168 6387 raft_consensus.cc:686] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 1 LEADER]: Becoming Leader. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Running, Role: LEADER
I20260620 07:11:41.057353 6387 consensus_queue.cc:237] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.059208 5778 catalog_manager.cc:5495] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d reported cstate change: term changed from 0 to 1, leader changed from <none> to b695340011e648d6802a4ed1d6bfef8d (127.3.245.2). New cstate: current_term: 1 leader_uuid: "b695340011e648d6802a4ed1d6bfef8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:41.082087 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:41.078519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.3.245.2:33983)
0620 07:11:41.078821 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:41.078829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:41.078842 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:11:41.078884 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:41.078886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:41.078929 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:41.079019 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:41.079296 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.079830 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.080873 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.081098 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.081197 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:41.081460 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:41.081534 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:41.081597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:41.081831 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 07:11:41.081891 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":142,"threads_started":1}
I20260620 07:11:41.088724 6215 raft_consensus.cc:1261] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Refusing update from remote peer b695340011e648d6802a4ed1d6bfef8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:41.088836 5936 raft_consensus.cc:1261] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Refusing update from remote peer b695340011e648d6802a4ed1d6bfef8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:41.089118 6388 consensus_queue.cc:1035] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:41.089263 6387 consensus_queue.cc:1035] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:41.121505 5778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:41.121662 5778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:41.123757 5778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Sending DeleteTablet for 3 replicas of tablet ad75eb79a10248aabf0523e21af41642
I20260620 07:11:41.124401 5916 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75eb79a10248aabf0523e21af41642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:41 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260620 07:11:41.124454 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75eb79a10248aabf0523e21af41642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:41 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 07:11:41.124473 6056 tablet_service.cc:1514] Processing DeleteTablet for tablet ad75eb79a10248aabf0523e21af41642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:41 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:45884
I20260620 07:11:41.124660 6371 tablet_replica.cc:330] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: stopping tablet replica
I20260620 07:11:41.124679 6370 tablet_replica.cc:330] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: stopping tablet replica
I20260620 07:11:41.124783 6371 raft_consensus.cc:2229] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:41.124792 6370 raft_consensus.cc:2229] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:41.124909 6370 raft_consensus.cc:2258] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:41.124909 6371 raft_consensus.cc:2258] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:41.125082 6372 tablet_replica.cc:330] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: stopping tablet replica
I20260620 07:11:41.125212 6372 raft_consensus.cc:2229] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:41.125432 6372 raft_consensus.cc:2258] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:41.125352 5778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:41.125790 5778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:41.125790 6370 ts_tablet_manager.cc:1905] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:41.126227 6372 ts_tablet_manager.cc:1905] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:41.126348 6371 ts_tablet_manager.cc:1905] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:41.128942 6372 ts_tablet_manager.cc:1918] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:41.128975 6370 ts_tablet_manager.cc:1918] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:41.129035 6372 log.cc:1196] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/ad75eb79a10248aabf0523e21af41642
I20260620 07:11:41.129057 6370 log.cc:1196] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/ad75eb79a10248aabf0523e21af41642
I20260620 07:11:41.129320 6372 ts_tablet_manager.cc:1939] T ad75eb79a10248aabf0523e21af41642 P b695340011e648d6802a4ed1d6bfef8d: Deleting consensus metadata
I20260620 07:11:41.129320 6370 ts_tablet_manager.cc:1939] T ad75eb79a10248aabf0523e21af41642 P 965f675d2b364d0b92a075833df35055: Deleting consensus metadata
I20260620 07:11:41.129346 6371 ts_tablet_manager.cc:1918] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:11:41.129420 6371 log.cc:1196] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/ad75eb79a10248aabf0523e21af41642
I20260620 07:11:41.129690 6371 ts_tablet_manager.cc:1939] T ad75eb79a10248aabf0523e21af41642 P a27b35b586c84071b98a02f36caf28a0: Deleting consensus metadata
I20260620 07:11:41.130036 5763 catalog_manager.cc:4841] TS 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819): tablet ad75eb79a10248aabf0523e21af41642 (table test-table [id=fb310972a4d74fe9a610131286dfff93]) successfully deleted
I20260620 07:11:41.130234 5765 catalog_manager.cc:4841] TS b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983): tablet ad75eb79a10248aabf0523e21af41642 (table test-table [id=fb310972a4d74fe9a610131286dfff93]) successfully deleted
I20260620 07:11:41.130502 5763 catalog_manager.cc:4841] TS a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065): tablet ad75eb79a10248aabf0523e21af41642 (table test-table [id=fb310972a4d74fe9a610131286dfff93]) successfully deleted
I20260620 07:11:41.131325 6056 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1f56e8167747f5884bce97cf56593d (DEFAULT_TABLE table=test-table [id=e6e4deacd47f4a4887f41dec45d3cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:41.131325 5916 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1f56e8167747f5884bce97cf56593d (DEFAULT_TABLE table=test-table [id=e6e4deacd47f4a4887f41dec45d3cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:41.131472 6196 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1f56e8167747f5884bce97cf56593d (DEFAULT_TABLE table=test-table [id=e6e4deacd47f4a4887f41dec45d3cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:41.131575 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1f56e8167747f5884bce97cf56593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:41.131575 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1f56e8167747f5884bce97cf56593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:41.131716 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c1f56e8167747f5884bce97cf56593d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:41.133496 6383 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Bootstrap starting.
I20260620 07:11:41.133948 6381 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Bootstrap starting.
I20260620 07:11:41.133948 6382 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055: Bootstrap starting.
I20260620 07:11:41.134382 6383 tablet_bootstrap.cc:654] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:41.134686 6382 tablet_bootstrap.cc:654] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:41.134791 6381 tablet_bootstrap.cc:654] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:41.135672 6383 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: No bootstrap required, opened a new log
I20260620 07:11:41.135774 6383 ts_tablet_manager.cc:1397] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:41.135818 6381 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: No bootstrap required, opened a new log
I20260620 07:11:41.135918 6381 ts_tablet_manager.cc:1397] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:41.136250 6383 raft_consensus.cc:348] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.136346 6383 raft_consensus.cc:374] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:41.136368 6383 raft_consensus.cc:729] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27b35b586c84071b98a02f36caf28a0, State: Initialized, Role: FOLLOWER
I20260620 07:11:41.136304 6381 raft_consensus.cc:348] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.136435 6381 raft_consensus.cc:374] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:41.136477 6381 raft_consensus.cc:729] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b695340011e648d6802a4ed1d6bfef8d, State: Initialized, Role: FOLLOWER
I20260620 07:11:41.136499 6383 consensus_queue.cc:260] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.136652 6381 consensus_queue.cc:260] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.136806 6383 ts_tablet_manager.cc:1428] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:11:41.136934 6382 tablet_bootstrap.cc:492] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055: No bootstrap required, opened a new log
I20260620 07:11:41.136956 6381 ts_tablet_manager.cc:1428] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:41.137018 6382 ts_tablet_manager.cc:1397] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:41.137365 6382 raft_consensus.cc:348] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.137490 6382 raft_consensus.cc:374] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:41.137528 6382 raft_consensus.cc:729] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965f675d2b364d0b92a075833df35055, State: Initialized, Role: FOLLOWER
I20260620 07:11:41.137656 6382 consensus_queue.cc:260] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.137944 6382 ts_tablet_manager.cc:1428] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:11:41.208247 6356 raft_consensus.cc:482] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:41.208401 6356 raft_consensus.cc:504] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.209012 6356 leader_election.cc:288] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:41.209606 6215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1f56e8167747f5884bce97cf56593d" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0" is_pre_election: true
I20260620 07:11:41.209606 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1f56e8167747f5884bce97cf56593d" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d" is_pre_election: true
I20260620 07:11:41.209762 6215 raft_consensus.cc:2454] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 0.
I20260620 07:11:41.209762 6076 raft_consensus.cc:2454] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 0.
I20260620 07:11:41.210094 5867 leader_election.cc:302] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, a27b35b586c84071b98a02f36caf28a0; no voters:
I20260620 07:11:41.210367 6356 raft_consensus.cc:2781] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:41.210448 6356 raft_consensus.cc:482] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:41.210500 6356 raft_consensus.cc:3037] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.211634 6356 raft_consensus.cc:504] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.212059 6356 leader_election.cc:288] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 election: Requested vote from peers b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983), a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065)
I20260620 07:11:41.212505 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1f56e8167747f5884bce97cf56593d" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b695340011e648d6802a4ed1d6bfef8d"
I20260620 07:11:41.212505 6215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1f56e8167747f5884bce97cf56593d" candidate_uuid: "965f675d2b364d0b92a075833df35055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a27b35b586c84071b98a02f36caf28a0"
I20260620 07:11:41.212657 6076 raft_consensus.cc:3037] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.212657 6215 raft_consensus.cc:3037] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:41.213887 6076 raft_consensus.cc:2454] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 1.
I20260620 07:11:41.213887 6215 raft_consensus.cc:2454] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 965f675d2b364d0b92a075833df35055 in term 1.
I20260620 07:11:41.214221 5869 leader_election.cc:302] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 965f675d2b364d0b92a075833df35055, b695340011e648d6802a4ed1d6bfef8d; no voters:
I20260620 07:11:41.214604 6356 raft_consensus.cc:2781] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:41.214740 6356 raft_consensus.cc:686] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [term 1 LEADER]: Becoming Leader. State: Replica: 965f675d2b364d0b92a075833df35055, State: Running, Role: LEADER
I20260620 07:11:41.214887 6356 consensus_queue.cc:237] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } }
I20260620 07:11:41.216543 5777 catalog_manager.cc:5495] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 965f675d2b364d0b92a075833df35055 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "965f675d2b364d0b92a075833df35055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965f675d2b364d0b92a075833df35055" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:41.248123 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:41.244422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38870 (local address 127.3.245.1:37819)
0620 07:11:41.244833 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:41.244842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:41.244854 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:11:41.244893 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:41.244895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:41.244942 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:41.245020 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:41.245255 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.246073 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.246860 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.247229 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.247330 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:41.247611 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:41.247698 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:41.247767 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:41.247911 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 07:11:41.247967 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":132,"threads_started":1}
I20260620 07:11:41.258123 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:41.254240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.3.245.1:37313)
0620 07:11:41.254552 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:41.254564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:41.254652 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:41.254894 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:41.254896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:41.254902 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:11:41.255009 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:41.255014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.256050 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.256053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:41.256803 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:41.256813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.257437 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.257440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:41.257520 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:41.257547 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:11:41.257845 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:11:41.257851 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:11:41.257906 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 07:11:41.257977 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260620 07:11:41.258124 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:41.254401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:37313 (local address 127.3.245.1:37819)
0620 07:11:41.254501 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:41.254505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:41.254640 (+ 135us) server_negotiation.cc:407] Connection header received
0620 07:11:41.254735 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:41.254738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:41.254785 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:41.254855 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:41.255105 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.255885 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.256922 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:41.257269 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:41.257400 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:41.257648 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:41.257728 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:41.257794 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:41.257935 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 07:11:41.257990 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":27}
I20260620 07:11:41.264133 6215 raft_consensus.cc:1261] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Refusing update from remote peer 965f675d2b364d0b92a075833df35055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:41.264166 6076 raft_consensus.cc:1261] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Refusing update from remote peer 965f675d2b364d0b92a075833df35055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:41.264488 6325 consensus_queue.cc:1035] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a27b35b586c84071b98a02f36caf28a0" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:41.264674 6356 consensus_queue.cc:1035] T 5c1f56e8167747f5884bce97cf56593d P 965f675d2b364d0b92a075833df35055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b695340011e648d6802a4ed1d6bfef8d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:11:41.279044 6349 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 07:11:42.835078 5779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:42.835304 5779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:42.837802 5779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde307b863b84711907aca22cd35726a: Sending DeleteTablet for 3 replicas of tablet 5c1f56e8167747f5884bce97cf56593d
I20260620 07:11:42.838472 6056 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1f56e8167747f5884bce97cf56593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:42 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:45884
I20260620 07:11:42.838495 5915 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1f56e8167747f5884bce97cf56593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:42 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:55218
I20260620 07:11:42.838529 6196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c1f56e8167747f5884bce97cf56593d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:42 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:49668
I20260620 07:11:42.838850 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5851
I20260620 07:11:42.839819 6427 tablet_replica.cc:330] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: stopping tablet replica
I20260620 07:11:42.840000 6427 raft_consensus.cc:2229] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:42.840124 6427 raft_consensus.cc:2258] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:42.840749 6429 tablet_replica.cc:330] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: stopping tablet replica
I20260620 07:11:42.840873 6427 ts_tablet_manager.cc:1905] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:42.840915 6429 raft_consensus.cc:2229] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:42.841043 6429 raft_consensus.cc:2258] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:42.841640 6429 ts_tablet_manager.cc:1905] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:42.843961 6429 ts_tablet_manager.cc:1918] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:11:42.844060 6429 log.cc:1196] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/5c1f56e8167747f5884bce97cf56593d
I20260620 07:11:42.844383 6429 ts_tablet_manager.cc:1939] T 5c1f56e8167747f5884bce97cf56593d P a27b35b586c84071b98a02f36caf28a0: Deleting consensus metadata
I20260620 07:11:42.844635 6427 ts_tablet_manager.cc:1918] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 07:11:42.844763 6427 log.cc:1196] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/5c1f56e8167747f5884bce97cf56593d
I20260620 07:11:42.845054 6427 ts_tablet_manager.cc:1939] T 5c1f56e8167747f5884bce97cf56593d P b695340011e648d6802a4ed1d6bfef8d: Deleting consensus metadata
I20260620 07:11:42.845288 5763 catalog_manager.cc:4841] TS a27b35b586c84071b98a02f36caf28a0 (127.3.245.3:38065): tablet 5c1f56e8167747f5884bce97cf56593d (table test-table [id=e6e4deacd47f4a4887f41dec45d3cece]) successfully deleted
I20260620 07:11:42.845873 5765 catalog_manager.cc:4841] TS b695340011e648d6802a4ed1d6bfef8d (127.3.245.2:33983): tablet 5c1f56e8167747f5884bce97cf56593d (table test-table [id=e6e4deacd47f4a4887f41dec45d3cece]) successfully deleted
I20260620 07:11:42.848462 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5991
W20260620 07:11:42.848613 5763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37819 (error 108)
I20260620 07:11:42.853674 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:42.853238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37819 (local address 127.0.0.1:38880)
0620 07:11:42.853513 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:42.853583 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
W20260620 07:11:42.853826 5763 catalog_manager.cc:4584] TS 965f675d2b364d0b92a075833df35055 (127.3.245.1:37819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c1f56e8167747f5884bce97cf56593d: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37819: connect: Connection refused (error 111)
I20260620 07:11:42.856269 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6131
I20260620 07:11:42.863478 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 5747
2026-06-20T07:11:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6562 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:42 dist-test-slave-ldfr krb5kdc[6434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:42 dist-test-slave-ldfr krb5kdc[6434](info): set up 2 sockets
Jun 20 07:11:42 dist-test-slave-ldfr krb5kdc[6434](info): commencing operation
krb5kdc: starting...
W20260620 07:11:43.936939 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:44.064438 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39637
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:39637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:44.175479 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:44.175774 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:44.175817 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:44.179993 6447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:44.180058 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:44.180078 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:44.180095 6447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:44.180111 6447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:44.185575 6447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:39637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:44.186689 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:44.187800 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:44.196893 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:44.196880 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:44.196933 6447 server_base.cc:1034] running on GCE node
W20260620 07:11:44.196877 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:44.197781 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:44.198918 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:44.200116 6447 hybrid_clock.cc:648] HybridClock initialized: now 1781939504200106 us; error 32 us; skew 500 ppm
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:44.203068 6447 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:44.203974 6447 webserver.cc:460] Webserver started at http://127.3.245.62:40493/ using document root <none> and password file <none>
I20260620 07:11:44.204543 6447 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:44.204653 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:44.204893 6447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:44.207216 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "269e7342f8a241eeb9f4950fbe8222a3"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "e8f6aca912dada8dbe873d5b728d40ce"
server_key_iv: "3912fe228cd984579892f82aae2ba841"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.207894 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "269e7342f8a241eeb9f4950fbe8222a3"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "e8f6aca912dada8dbe873d5b728d40ce"
server_key_iv: "3912fe228cd984579892f82aae2ba841"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.211776 6447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 07:11:44.214442 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.215809 6447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:44.215973 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "269e7342f8a241eeb9f4950fbe8222a3"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "e8f6aca912dada8dbe873d5b728d40ce"
server_key_iv: "3912fe228cd984579892f82aae2ba841"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.216080 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:44.251832 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:44.255336 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:44.255568 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:44.263360 6447 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:39637
I20260620 07:11:44.263372 6514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:39637 every 8 connection(s)
I20260620 07:11:44.264362 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:44.267243 6515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:44.271317 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6447
I20260620 07:11:44.271411 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:44.271680 4052 external_mini_cluster.cc:1428] Setting key c2dc868338f0f0a794ad177158a76ae4
I20260620 07:11:44.273197 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: Bootstrap starting.
I20260620 07:11:44.275993 6515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:44.276928 6515 log.cc:824] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:44.278975 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: No bootstrap required, opened a new log
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:44.282748 6515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } }
I20260620 07:11:44.283026 6515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:44.283107 6515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269e7342f8a241eeb9f4950fbe8222a3, State: Initialized, Role: FOLLOWER
I20260620 07:11:44.283614 6515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } }
I20260620 07:11:44.283995 6515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:44.284209 6515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:44.284353 6515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:44.285378 6515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } }
I20260620 07:11:44.285849 6515 leader_election.cc:302] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 269e7342f8a241eeb9f4950fbe8222a3; no voters:
I20260620 07:11:44.286238 6515 leader_election.cc:288] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:44.286378 6520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:44.286572 6520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [term 1 LEADER]: Becoming Leader. State: Replica: 269e7342f8a241eeb9f4950fbe8222a3, State: Running, Role: LEADER
I20260620 07:11:44.286952 6520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } }
I20260620 07:11:44.287364 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:44.273151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.3.245.62:39637)
0620 07:11:44.273607 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:44.273626 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:44.273668 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:11:44.274551 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:44.274576 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:44.275025 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:44.275378 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:44.276054 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.277495 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.278760 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.279369 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.279546 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:44.282106 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:44.282111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:44.282131 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:44.282158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:44.284089 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.284644 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.284650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.284658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.284772 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.285064 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.285068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.285071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.285478 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:44.285671 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:44.285979 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 07:11:44.286330 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260620 07:11:44.287611 6515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:44.289230 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 269e7342f8a241eeb9f4950fbe8222a3. Latest consensus state: current_term: 1 leader_uuid: "269e7342f8a241eeb9f4950fbe8222a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } } }
I20260620 07:11:44.289350 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:44.289690 6522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "269e7342f8a241eeb9f4950fbe8222a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "269e7342f8a241eeb9f4950fbe8222a3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39637 } } }
I20260620 07:11:44.289824 6522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3 [sys.catalog]: This master's current role is: LEADER
W20260620 07:11:44.292552 6535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:44.292680 6535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:44.292773 6533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:44.293689 6533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:44.299607 6533 catalog_manager.cc:1347] Generated new cluster ID: 98f76e007197486691f8c4019011b346
I20260620 07:11:44.299696 6533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:44.357534 6533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:44.358578 6533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:44.370783 6533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 269e7342f8a241eeb9f4950fbe8222a3: Generated new TSK 0
I20260620 07:11:44.371507 6533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:44.413872 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:44.537230 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:44.537501 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:44.537545 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:44.541674 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:44.541736 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:44.541828 6541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:44.547230 6541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:44.548405 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:44.549532 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:44.560052 6547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:44.560050 6549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:44.560050 6546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:44.560518 6541 server_base.cc:1034] running on GCE node
I20260620 07:11:44.560976 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:44.561678 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:44.562855 6541 hybrid_clock.cc:648] HybridClock initialized: now 1781939504562823 us; error 47 us; skew 500 ppm
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:44.565994 6541 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:44.567006 6541 webserver.cc:460] Webserver started at http://127.3.245.1:39333/ using document root <none> and password file <none>
I20260620 07:11:44.567611 6541 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:44.567706 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:44.567950 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:44.570109 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "d6ea4e76299044f6af701215b91d0d1b"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "57087c81c07b15dda775699f5cd9a091"
server_key_iv: "96e3566b3206269268997755fed84554"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.570713 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "d6ea4e76299044f6af701215b91d0d1b"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "57087c81c07b15dda775699f5cd9a091"
server_key_iv: "96e3566b3206269268997755fed84554"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.574540 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:11:44.577179 6556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.578548 6541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:44.578750 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "d6ea4e76299044f6af701215b91d0d1b"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "57087c81c07b15dda775699f5cd9a091"
server_key_iv: "96e3566b3206269268997755fed84554"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.578874 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:44.596491 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:44.599906 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:44.600140 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:44.600833 6541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:44.601718 6541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:44.601769 6541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.601840 6541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:44.601883 6541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.611953 6541 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:39861
I20260620 07:11:44.611975 6669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:39861 every 8 connection(s)
I20260620 07:11:44.612972 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:44.621088 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6541
I20260620 07:11:44.621186 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:44.621469 4052 external_mini_cluster.cc:1428] Setting key 7d2256abea513ff78d5f43b576f38abb
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:44.628273 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:44.614950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:33023 (local address 127.3.245.62:39637)
0620 07:11:44.615126 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:44.615131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:44.616107 (+ 976us) server_negotiation.cc:407] Connection header received
0620 07:11:44.617782 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:44.617785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:44.617843 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:44.617925 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:44.619325 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.619946 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.621370 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.621676 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.621790 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:44.624292 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:44.624296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:44.624298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:44.624324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:44.625826 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.626470 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.626473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.626474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.626552 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.626942 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.626944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.626945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.627117 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:44.627204 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:44.627992 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 07:11:44.628126 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:11:44.629528 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:44.615406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39637 (local address 127.3.245.1:33023)
0620 07:11:44.615921 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:44.615972 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:44.617537 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:44.618135 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:44.618150 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:44.618608 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:44.619156 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:44.619172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.620106 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.620109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:44.621238 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:44.621249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.621838 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.621843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:44.622016 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:44.622643 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:44.622667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:44.624130 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:44.625961 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:44.625970 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:44.625990 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:44.626365 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:44.626646 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:44.626652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:44.626657 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:44.626845 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:44.627236 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:44.627246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:44.627626 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 07:11:44.627875 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:11:44.628261 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260620 07:11:44.630909 6670 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39637
I20260620 07:11:44.631192 6670 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:11:44.631824 6670 heartbeater.cc:511] Master 127.3.245.62:39637 requested a full tablet report, sending...
I20260620 07:11:44.633602 6479 ts_manager.cc:194] Registered new tserver with Master: d6ea4e76299044f6af701215b91d0d1b (127.3.245.1:39861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:11:44.634987 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:33023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:44.652307 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:44.761673 6675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:44.761998 6675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:44.762082 6675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:44.766161 6675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:44.766265 6675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:44.766359 6675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:44.771715 6675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:44.772804 6675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:44.773814 6675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:44.783494 6683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:44.783519 6681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:44.783789 6675 server_base.cc:1034] running on GCE node
W20260620 07:11:44.783816 6680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:44.784349 6675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:44.785138 6675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:44.786290 6675 hybrid_clock.cc:648] HybridClock initialized: now 1781939504786268 us; error 35 us; skew 500 ppm
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:44.789535 6675 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:44.790539 6675 webserver.cc:460] Webserver started at http://127.3.245.2:40573/ using document root <none> and password file <none>
I20260620 07:11:44.791137 6675 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:44.791239 6675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:44.791469 6675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:44.793596 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "b7cc7be74cf34f64b53b0cbdb3cbd199"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "f043a47863a736e2a182b66b41727fcf"
server_key_iv: "f0b118aee0e7f1fe4fd6ce373f428524"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.794147 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "b7cc7be74cf34f64b53b0cbdb3cbd199"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "f043a47863a736e2a182b66b41727fcf"
server_key_iv: "f0b118aee0e7f1fe4fd6ce373f428524"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.798280 6675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:44.800953 6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.802219 6675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:11:44.802354 6675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "b7cc7be74cf34f64b53b0cbdb3cbd199"
format_stamp: "Formatted at 2026-06-20 07:11:44 on dist-test-slave-ldfr"
server_key: "f043a47863a736e2a182b66b41727fcf"
server_key_iv: "f0b118aee0e7f1fe4fd6ce373f428524"
server_key_version: "encryptionkey@0"
I20260620 07:11:44.802470 6675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:44.832538 6675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:44.836026 6675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:44.836244 6675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:44.836905 6675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:44.837739 6675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:44.837806 6675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.837879 6675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:44.837931 6675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:44.847034 6675 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44933
I20260620 07:11:44.847082 6803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44933 every 8 connection(s)
I20260620 07:11:44.847965 6675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:44.849148 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6675
I20260620 07:11:44.849265 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:44.849527 4052 external_mini_cluster.cc:1428] Setting key da698e52498d1cc88ba89c416b5855e5
Jun 20 07:11:44 dist-test-slave-ldfr krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:11:44.862905 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:44.850123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:39049 (local address 127.3.245.62:39637)
0620 07:11:44.850282 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:44.850286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:44.851203 (+ 917us) server_negotiation.cc:407] Connection header received
0620 07:11:44.852683 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:44.852686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:44.852775 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:44.852884 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:44.854052 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.854581 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.855714 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.856113 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.856243 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:44.858969 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:44.858976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:44.858981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:44.859011 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:44.860809 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.861417 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.861423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.861425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.861520 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:44.861859 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:44.861862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:44.861863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:44.862047 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:44.862123 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:44.862660 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 07:11:44.862776 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:11:44.863935 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:44.850602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39637 (local address 127.3.245.2:39049)
0620 07:11:44.851059 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:44.851112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:44.852413 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:44.853078 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:44.853086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:44.853436 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:44.853910 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:44.853924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.854705 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.854717 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:44.855587 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:44.855596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:44.856282 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:44.856285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:44.856433 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:44.857220 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:44.857247 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:44.858791 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:44.860964 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:44.860970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:44.860983 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:44.861256 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:44.861622 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:44.861624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:44.861626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:44.861771 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:44.862171 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:44.862176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:44.862431 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 07:11:44.862594 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 07:11:44.862912 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260620 07:11:44.865386 6804 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39637
I20260620 07:11:44.865661 6804 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:44.866204 6804 heartbeater.cc:511] Master 127.3.245.62:39637 requested a full tablet report, sending...
I20260620 07:11:44.867414 6479 ts_manager.cc:194] Registered new tserver with Master: b7cc7be74cf34f64b53b0cbdb3cbd199 (127.3.245.2:44933)
I20260620 07:11:44.868019 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:39049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:44.881021 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:44.993361 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:44.993661 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:44.993749 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:44.997846 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:44.997934 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:44.998020 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:45.003798 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:45.005007 6809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:45.006111 6809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:45.016300 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:45.016299 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:45.016319 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:45.016667 6809 server_base.cc:1034] running on GCE node
I20260620 07:11:45.017153 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:45.017871 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:45.019142 6809 hybrid_clock.cc:648] HybridClock initialized: now 1781939505019106 us; error 50 us; skew 500 ppm
Jun 20 07:11:45 dist-test-slave-ldfr krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:45.022205 6809 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:45.023123 6809 webserver.cc:460] Webserver started at http://127.3.245.3:37003/ using document root <none> and password file <none>
I20260620 07:11:45.023770 6809 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:45.023871 6809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:45.024118 6809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:45.026360 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "334a1aca9cb04da8a0beef9454c6b83d"
format_stamp: "Formatted at 2026-06-20 07:11:45 on dist-test-slave-ldfr"
server_key: "ecb8f0e7ebe23ff82e6219e9a8dbd109"
server_key_iv: "9bda7c086c4e293e52e5e5817e3841e3"
server_key_version: "encryptionkey@0"
I20260620 07:11:45.026949 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "334a1aca9cb04da8a0beef9454c6b83d"
format_stamp: "Formatted at 2026-06-20 07:11:45 on dist-test-slave-ldfr"
server_key: "ecb8f0e7ebe23ff82e6219e9a8dbd109"
server_key_iv: "9bda7c086c4e293e52e5e5817e3841e3"
server_key_version: "encryptionkey@0"
I20260620 07:11:45.030937 6809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 07:11:45.033566 6824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:45.034951 6809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:45.035101 6809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "334a1aca9cb04da8a0beef9454c6b83d"
format_stamp: "Formatted at 2026-06-20 07:11:45 on dist-test-slave-ldfr"
server_key: "ecb8f0e7ebe23ff82e6219e9a8dbd109"
server_key_iv: "9bda7c086c4e293e52e5e5817e3841e3"
server_key_version: "encryptionkey@0"
I20260620 07:11:45.035230 6809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:45.053762 6809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:45.057118 6809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:45.057333 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:45.057930 6809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:45.058773 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:45.058843 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:45.058914 6809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:45.058964 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:45.067896 6809 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:44599
I20260620 07:11:45.067919 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:44599 every 8 connection(s)
I20260620 07:11:45.068885 6809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:45.078068 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6809
I20260620 07:11:45.078202 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:45.078491 4052 external_mini_cluster.cc:1428] Setting key c692dacdc1c815d2044833c382f1fb23
Jun 20 07:11:45 dist-test-slave-ldfr krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:45.083842 6518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:45.070689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:56745 (local address 127.3.245.62:39637)
0620 07:11:45.070850 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:45.070854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:45.071774 (+ 920us) server_negotiation.cc:407] Connection header received
0620 07:11:45.073352 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:45.073355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:45.073406 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:45.073483 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:45.074793 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:45.075376 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:45.076889 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:45.077172 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:45.077273 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:45.080029 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:45.080033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:45.080036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:45.080063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:45.081723 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:45.082266 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:45.082277 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:45.082279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:45.082353 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:45.082713 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:45.082718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:45.082720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:45.082912 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:45.083028 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:45.083559 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 07:11:45.083680 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:11:45.084769 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:45.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39637 (local address 127.3.245.3:56745)
0620 07:11:45.071558 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:45.071654 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:45.073093 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:45.073639 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:45.073647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:45.074060 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:45.074607 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:45.074623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:45.075522 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:45.075525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:45.076717 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:45.076728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:45.077342 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:45.077345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:45.077509 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:45.078302 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:45.078330 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:45.079873 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:45.081849 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:45.081856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:45.081872 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:45.082154 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:45.082446 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:45.082450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:45.082452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:45.082614 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:45.083024 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:45.083031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:45.083316 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 07:11:45.083518 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 07:11:45.083870 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260620 07:11:45.085872 6938 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39637
I20260620 07:11:45.086122 6938 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:45.086643 6938 heartbeater.cc:511] Master 127.3.245.62:39637 requested a full tablet report, sending...
I20260620 07:11:45.087726 6479 ts_manager.cc:194] Registered new tserver with Master: 334a1aca9cb04da8a0beef9454c6b83d (127.3.245.3:44599)
I20260620 07:11:45.088402 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:56745
I20260620 07:11:45.092597 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:11:45.106417 6518 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:11:45.104741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.3.245.62:39637)
0620 07:11:45.104952 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:45.104958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:45.105115 (+ 157us) server_negotiation.cc:407] Connection header received
0620 07:11:45.105294 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:45.105296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:45.105340 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:45.105456 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:45.106250 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43442: BlockingRecv error: recv got EOF from 127.0.0.1:43442 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:11:45.107667 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6541
I20260620 07:11:45.113502 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6675
I20260620 07:11:45.118687 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6809
I20260620 07:11:45.124208 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6447
2026-06-20T07:11:45Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2259 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:45 dist-test-slave-ldfr krb5kdc[6950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:45 dist-test-slave-ldfr krb5kdc[6950](info): set up 2 sockets
Jun 20 07:11:45 dist-test-slave-ldfr krb5kdc[6950](info): commencing operation
krb5kdc: starting...
W20260620 07:11:46.176841 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[6950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:46.298156 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39385
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:39385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:46.410719 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:46.411118 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:46.411207 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:46.415366 6963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:46.415457 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:46.415493 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:46.415535 6963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:46.415571 6963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:46.421018 6963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:39385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:46.422114 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:46.423187 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:46.432427 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:46.432582 6963 server_base.cc:1034] running on GCE node
W20260620 07:11:46.432461 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:46.432473 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:46.433266 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:46.434501 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:46.435750 6963 hybrid_clock.cc:648] HybridClock initialized: now 1781939506435720 us; error 45 us; skew 500 ppm
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[6950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:46.443796 6963 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:46.444945 6963 webserver.cc:460] Webserver started at http://127.3.245.62:41929/ using document root <none> and password file <none>
I20260620 07:11:46.445577 6963 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:46.445678 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:46.445914 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:46.448117 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f"
format_stamp: "Formatted at 2026-06-20 07:11:46 on dist-test-slave-ldfr"
server_key: "d44b449ebb8238ffe8ffebdb755cb420"
server_key_iv: "d327c05b8cdb09ba5a2475d129cd9619"
server_key_version: "encryptionkey@0"
I20260620 07:11:46.448881 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f"
format_stamp: "Formatted at 2026-06-20 07:11:46 on dist-test-slave-ldfr"
server_key: "d44b449ebb8238ffe8ffebdb755cb420"
server_key_iv: "d327c05b8cdb09ba5a2475d129cd9619"
server_key_version: "encryptionkey@0"
I20260620 07:11:46.453639 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:11:46.456835 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:46.458158 6963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:11:46.458304 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f"
format_stamp: "Formatted at 2026-06-20 07:11:46 on dist-test-slave-ldfr"
server_key: "d44b449ebb8238ffe8ffebdb755cb420"
server_key_iv: "d327c05b8cdb09ba5a2475d129cd9619"
server_key_version: "encryptionkey@0"
I20260620 07:11:46.458432 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:46.495019 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:46.498513 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:46.498729 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:46.506179 6963 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:39385
I20260620 07:11:46.506467 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:39385 every 8 connection(s)
I20260620 07:11:46.507205 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:46.510155 7031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:46.515617 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 6963
I20260620 07:11:46.515725 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:46.515992 4052 external_mini_cluster.cc:1428] Setting key fe616eb491a812d5c2d5c1f15f769e0a
I20260620 07:11:46.516211 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: Bootstrap starting.
I20260620 07:11:46.518810 7031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:46.519804 7031 log.cc:824] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:46.522292 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: No bootstrap required, opened a new log
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[6950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:46.527120 7031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } }
I20260620 07:11:46.527458 7031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:46.527570 7031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d16a591d8e540a7ba2b75c3cedbfc6f, State: Initialized, Role: FOLLOWER
I20260620 07:11:46.528064 7031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } }
I20260620 07:11:46.528196 7031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:46.528295 7031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:46.528416 7031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:46.529480 7031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } }
I20260620 07:11:46.529870 7031 leader_election.cc:302] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d16a591d8e540a7ba2b75c3cedbfc6f; no voters:
I20260620 07:11:46.530246 7031 leader_election.cc:288] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:46.530335 7036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:46.530553 7036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [term 1 LEADER]: Becoming Leader. State: Replica: 7d16a591d8e540a7ba2b75c3cedbfc6f, State: Running, Role: LEADER
I20260620 07:11:46.530874 7036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } }
I20260620 07:11:46.531687 7031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:46.532162 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:46.517352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.3.245.62:39385)
0620 07:11:46.517787 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:46.517803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:46.517840 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:11:46.518480 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:46.518497 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:46.518847 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:46.519184 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:46.520118 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:46.521315 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:46.522534 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:46.523066 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:46.523200 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:46.526409 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:46.526415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:46.526433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:46.526467 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:46.528591 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:46.529085 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:46.529091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:46.529097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:46.529223 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:46.529517 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:46.529520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:46.529522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:46.529990 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:46.530212 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:46.530914 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 07:11:46.531225 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260620 07:11:46.532541 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d16a591d8e540a7ba2b75c3cedbfc6f. Latest consensus state: current_term: 1 leader_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } } }
I20260620 07:11:46.532729 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:46.533113 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d16a591d8e540a7ba2b75c3cedbfc6f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39385 } } }
I20260620 07:11:46.533223 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:46.535663 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 07:11:46.537693 7052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:46.537787 7052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:46.537860 7045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:46.538527 7045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:46.544205 7045 catalog_manager.cc:1347] Generated new cluster ID: bcf80400a3604211a449186e121d8ed0
I20260620 07:11:46.544271 7045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:46.557102 7045 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:46.558123 7045 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:46.568710 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:46.562619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38922 (local address 127.3.245.62:39385)
0620 07:11:46.562731 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:46.562735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:46.563094 (+ 359us) server_negotiation.cc:407] Connection header received
0620 07:11:46.563245 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:46.563248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:46.563308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:46.563408 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:46.563946 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:46.564820 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:46.566007 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:46.566384 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:46.566498 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:46.567451 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:46.567455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:46.567459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:46.568207 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:46.568243 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:46.568263 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:46.568367 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:46.568466 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:11:46.568533 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:11:46.577669 7045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d16a591d8e540a7ba2b75c3cedbfc6f: Generated new TSK 0
I20260620 07:11:46.578511 7045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:11:46.591101 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 6963
2026-06-20T07:11:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1464 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[7064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[7064](info): set up 2 sockets
Jun 20 07:11:46 dist-test-slave-ldfr krb5kdc[7064](info): commencing operation
krb5kdc: starting...
W20260620 07:11:47.640836 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:47 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:47.767225 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:33005
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:47.881585 7077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:47.881908 7077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:47.881997 7077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:47.886152 7077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:47.886241 7077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:47.886281 7077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:47.886329 7077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:47.886363 7077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:47.891744 7077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:47.892917 7077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:47.893960 7077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:47.903015 7082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:47.903015 7083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:47.903009 7085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:47.903512 7077 server_base.cc:1034] running on GCE node
I20260620 07:11:47.904111 7077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:47.905324 7077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:47.906517 7077 hybrid_clock.cc:648] HybridClock initialized: now 1781939507906473 us; error 55 us; skew 500 ppm
Jun 20 07:11:47 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:47.909368 7077 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:47.910243 7077 webserver.cc:460] Webserver started at http://127.3.245.62:39603/ using document root <none> and password file <none>
I20260620 07:11:47.910786 7077 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:47.910841 7077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:47.911005 7077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:47.913116 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "66b05621ae724d3b9b9fd57f81c682e3"
format_stamp: "Formatted at 2026-06-20 07:11:47 on dist-test-slave-ldfr"
server_key: "54cef7bae4a767678c662f4afd0a2e62"
server_key_iv: "b56b903470b7c6619c31656be9c75e9d"
server_key_version: "encryptionkey@0"
I20260620 07:11:47.913729 7077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "66b05621ae724d3b9b9fd57f81c682e3"
format_stamp: "Formatted at 2026-06-20 07:11:47 on dist-test-slave-ldfr"
server_key: "54cef7bae4a767678c662f4afd0a2e62"
server_key_iv: "b56b903470b7c6619c31656be9c75e9d"
server_key_version: "encryptionkey@0"
I20260620 07:11:47.917663 7077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:11:47.920159 7092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:47.921551 7077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:47.921698 7077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "66b05621ae724d3b9b9fd57f81c682e3"
format_stamp: "Formatted at 2026-06-20 07:11:47 on dist-test-slave-ldfr"
server_key: "54cef7bae4a767678c662f4afd0a2e62"
server_key_iv: "b56b903470b7c6619c31656be9c75e9d"
server_key_version: "encryptionkey@0"
I20260620 07:11:47.921813 7077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:47.933225 7077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:47.936488 7077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:47.936721 7077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:47.943826 7077 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:33005
I20260620 07:11:47.944187 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:33005 every 8 connection(s)
I20260620 07:11:47.944878 7077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:47.947624 7145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:47.952988 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3: Bootstrap starting.
I20260620 07:11:47.954057 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7077
I20260620 07:11:47.954196 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:47.954499 4052 external_mini_cluster.cc:1428] Setting key 7ee4dd90ce8d4d4da64c0560d7200448
I20260620 07:11:47.955651 7145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:47.956517 7145 log.cc:824] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:47.958707 7145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3: No bootstrap required, opened a new log
I20260620 07:11:47.962978 7145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } }
I20260620 07:11:47.963300 7145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:47.963392 7145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b05621ae724d3b9b9fd57f81c682e3, State: Initialized, Role: FOLLOWER
I20260620 07:11:47.963793 7145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } }
I20260620 07:11:47.963897 7145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Jun 20 07:11:47 dist-test-slave-ldfr krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:47.963936 7145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:47.964027 7145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:47.965099 7145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } }
I20260620 07:11:47.965430 7145 leader_election.cc:302] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66b05621ae724d3b9b9fd57f81c682e3; no voters:
I20260620 07:11:47.965725 7145 leader_election.cc:288] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:47.965909 7150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:47.966130 7150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [term 1 LEADER]: Becoming Leader. State: Replica: 66b05621ae724d3b9b9fd57f81c682e3, State: Running, Role: LEADER
I20260620 07:11:47.966451 7150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } }
I20260620 07:11:47.967038 7145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:47.967706 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66b05621ae724d3b9b9fd57f81c682e3. Latest consensus state: current_term: 1 leader_uuid: "66b05621ae724d3b9b9fd57f81c682e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } } }
I20260620 07:11:47.967828 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:47.968155 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66b05621ae724d3b9b9fd57f81c682e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66b05621ae724d3b9b9fd57f81c682e3" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 33005 } } }
I20260620 07:11:47.968243 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:47.968703 7157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:47.970790 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:47.955631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.3.245.62:33005)
0620 07:11:47.956111 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:47.956132 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:47.956172 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:11:47.956998 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:47.957056 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:47.957471 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:47.957773 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:47.958631 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:47.959766 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:47.960972 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:47.961479 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:47.961612 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:47.964595 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:47.964601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:47.964680 (+ 79us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:47.964710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:47.966799 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:47.967340 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:47.967346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:47.967352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:47.967498 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:47.967828 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:47.967832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:47.967833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:47.968394 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:47.968635 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:47.969500 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 07:11:47.969822 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260620 07:11:47.971881 7157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:47.977526 7157 catalog_manager.cc:1347] Generated new cluster ID: 2d9ac796532e4d918d621210fe3ab361
I20260620 07:11:47.977605 7157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:48.008141 7157 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:48.009443 7157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:48.018767 7157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66b05621ae724d3b9b9fd57f81c682e3: Generated new TSK 0
I20260620 07:11:48.019608 7157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:48.056213 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:48.179041 7171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:48.179327 7171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:48.179373 7171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:48.183638 7171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:48.183710 7171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:48.183794 7171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:48.189311 7171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:48.190356 7171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:48.191377 7171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:48.201320 7177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:48.201306 7176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.201453 7171 server_base.cc:1034] running on GCE node
W20260620 07:11:48.201306 7179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.202147 7171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:48.202836 7171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:48.204005 7171 hybrid_clock.cc:648] HybridClock initialized: now 1781939508203968 us; error 49 us; skew 500 ppm
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:48.207041 7171 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:48.208197 7171 webserver.cc:460] Webserver started at http://127.3.245.1:34803/ using document root <none> and password file <none>
I20260620 07:11:48.208879 7171 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:48.208974 7171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:48.209231 7171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:48.211452 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "3a3a71b08b394e59aef5652c57e1a0b8"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "bbdf120072d078c0840b700010f644ef"
server_key_iv: "5a604b5fa077dcf671482cf765c2fa2d"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.212096 7171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "3a3a71b08b394e59aef5652c57e1a0b8"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "bbdf120072d078c0840b700010f644ef"
server_key_iv: "5a604b5fa077dcf671482cf765c2fa2d"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.216218 7171 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 07:11:48.219085 7186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.220563 7171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:48.220732 7171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "3a3a71b08b394e59aef5652c57e1a0b8"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "bbdf120072d078c0840b700010f644ef"
server_key_iv: "5a604b5fa077dcf671482cf765c2fa2d"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.220857 7171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:48.237982 7171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:48.241338 7171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:48.241541 7171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:48.242136 7171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:48.242974 7171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:48.243038 7171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.243103 7171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:48.243148 7171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.252005 7171 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:36679
I20260620 07:11:48.252030 7299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:36679 every 8 connection(s)
I20260620 07:11:48.252993 7171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:48.263276 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7171
I20260620 07:11:48.263412 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:48.263751 4052 external_mini_cluster.cc:1428] Setting key 91f5382a58fa52eaae215a2a3adc6ec5
I20260620 07:11:48.267333 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.254860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:55389 (local address 127.3.245.62:33005)
0620 07:11:48.254993 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:48.254997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:48.255842 (+ 845us) server_negotiation.cc:407] Connection header received
0620 07:11:48.257436 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:48.257439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:48.257560 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:48.257653 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:48.258836 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.259367 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.260506 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.260835 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.260943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.263474 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:48.263481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:48.263485 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:48.263514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:48.265127 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.265748 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.265752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.265754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.265836 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.266176 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.266181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.266183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.266359 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:48.266465 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:48.266987 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 07:11:48.267152 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:11:48.268229 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.255193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:33005 (local address 127.3.245.1:55389)
0620 07:11:48.255681 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:48.255727 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:48.257085 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:48.257826 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:48.257834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:48.258210 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:48.258688 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.258702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.259488 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.259493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.260378 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.260387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.260996 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.261000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.261157 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.261745 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:48.261761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:48.263260 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:48.265245 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.265251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.265266 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.265648 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.265932 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.265935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.265937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.266077 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.266463 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:48.266470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:48.266726 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 07:11:48.266916 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:11:48.267276 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260620 07:11:48.269531 7300 heartbeater.cc:347] Connected to a master server at 127.3.245.62:33005
I20260620 07:11:48.269783 7300 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:48.270275 7300 heartbeater.cc:511] Master 127.3.245.62:33005 requested a full tablet report, sending...
I20260620 07:11:48.271976 7109 ts_manager.cc:194] Registered new tserver with Master: 3a3a71b08b394e59aef5652c57e1a0b8 (127.3.245.1:36679)
I20260620 07:11:48.273675 7109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:55389
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:48.292272 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:48.405864 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:48.406175 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:48.406258 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:48.410377 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:48.410457 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:48.410543 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:48.415973 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:48.417138 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:48.418183 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:48.428383 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:48.428426 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.428689 7305 server_base.cc:1034] running on GCE node
W20260620 07:11:48.428825 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.429241 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:48.429966 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:48.431151 7305 hybrid_clock.cc:648] HybridClock initialized: now 1781939508431134 us; error 28 us; skew 500 ppm
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:48.434345 7305 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:48.435312 7305 webserver.cc:460] Webserver started at http://127.3.245.2:37375/ using document root <none> and password file <none>
I20260620 07:11:48.435897 7305 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:48.435957 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:48.436196 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:48.438333 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "bf70c650eb4f42d0bf8f099ec3b22d95"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "2c17fcb554b45ce3171bdde26ae10a7a"
server_key_iv: "d22a8e07eb9cc9e8a287d0cb7e016767"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.438922 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "bf70c650eb4f42d0bf8f099ec3b22d95"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "2c17fcb554b45ce3171bdde26ae10a7a"
server_key_iv: "d22a8e07eb9cc9e8a287d0cb7e016767"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.442826 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 07:11:48.445482 7320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.446977 7305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:48.447124 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "bf70c650eb4f42d0bf8f099ec3b22d95"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "2c17fcb554b45ce3171bdde26ae10a7a"
server_key_iv: "d22a8e07eb9cc9e8a287d0cb7e016767"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.447256 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:48.467044 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:48.470597 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:48.470813 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:48.471417 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:48.472250 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:48.472330 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.472383 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:48.472404 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.481549 7305 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:37841
I20260620 07:11:48.481565 7433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:37841 every 8 connection(s)
I20260620 07:11:48.482448 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:48.488873 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7305
I20260620 07:11:48.489001 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:48.489284 4052 external_mini_cluster.cc:1428] Setting key 063dd69f7e9e76c93d31f7c840cb2050
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:48.497704 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.484269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:33643 (local address 127.3.245.62:33005)
0620 07:11:48.484408 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:48.484411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:48.485245 (+ 834us) server_negotiation.cc:407] Connection header received
0620 07:11:48.486653 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:48.486656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:48.486714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:48.486814 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:48.487967 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.488503 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.490120 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.490402 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.490512 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.493453 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:48.493458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:48.493463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:48.493496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:48.495234 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.495792 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.495795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.495797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.495896 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.496366 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.496368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.496370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.496570 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:48.496726 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:48.497468 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 07:11:48.497579 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:48.498883 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.484656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:33005 (local address 127.3.245.2:33643)
0620 07:11:48.485104 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:48.485146 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:48.486403 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:48.486975 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:48.486981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:48.487352 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:48.487826 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.487839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.488646 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.488649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.489940 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.489950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.490626 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.490633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.490849 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.491524 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:48.491552 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:48.493274 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:48.495364 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.495372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.495391 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.495676 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.495995 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.495999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.496002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.496232 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.496745 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:48.496755 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:48.497141 (+ 386us) client_negotiation.cc:769] Sending connection context
0620 07:11:48.497382 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 07:11:48.497766 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260620 07:11:48.500108 7434 heartbeater.cc:347] Connected to a master server at 127.3.245.62:33005
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:11:48.500352 7434 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:48.500887 7434 heartbeater.cc:511] Master 127.3.245.62:33005 requested a full tablet report, sending...
I20260620 07:11:48.502020 7109 ts_manager.cc:194] Registered new tserver with Master: bf70c650eb4f42d0bf8f099ec3b22d95 (127.3.245.2:37841)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:11:48.502725 7109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:33643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:48.519188 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:48.637348 7439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:48.637637 7439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:48.637686 7439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:48.641812 7439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:48.641896 7439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:48.641995 7439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:48.647574 7439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:33005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:48.649020 7439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:48.650086 7439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:48.660162 7444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:48.660145 7445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.660259 7439 server_base.cc:1034] running on GCE node
W20260620 07:11:48.660145 7447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:48.660831 7439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:48.661486 7439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:48.662668 7439 hybrid_clock.cc:648] HybridClock initialized: now 1781939508662638 us; error 42 us; skew 500 ppm
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:48.665701 7439 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:48.666543 7439 webserver.cc:460] Webserver started at http://127.3.245.3:40667/ using document root <none> and password file <none>
I20260620 07:11:48.667203 7439 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:48.667290 7439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:48.667567 7439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:48.669742 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "241b9ef96f83493b90ca946c45620143"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "a2a86da0e9fa389ee64494cbad809887"
server_key_iv: "8893550ec4b76efdf628cb5eab25c570"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.670315 7439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "241b9ef96f83493b90ca946c45620143"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "a2a86da0e9fa389ee64494cbad809887"
server_key_iv: "8893550ec4b76efdf628cb5eab25c570"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.674089 7439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:48.676801 7454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.678241 7439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:11:48.678395 7439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "241b9ef96f83493b90ca946c45620143"
format_stamp: "Formatted at 2026-06-20 07:11:48 on dist-test-slave-ldfr"
server_key: "a2a86da0e9fa389ee64494cbad809887"
server_key_iv: "8893550ec4b76efdf628cb5eab25c570"
server_key_version: "encryptionkey@0"
I20260620 07:11:48.678517 7439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:48.694096 7439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:48.697661 7439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:48.697892 7439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:48.698542 7439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:48.699435 7439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:48.699509 7439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.699589 7439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:48.699635 7439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:48.709638 7439 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:36613
I20260620 07:11:48.709659 7567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:36613 every 8 connection(s)
I20260620 07:11:48.710717 7439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:48.716323 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7439
I20260620 07:11:48.716461 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:48.716781 4052 external_mini_cluster.cc:1428] Setting key 8882478ac3d012b4cc6ebee187aab2ad
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:48.726032 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.712733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:55325 (local address 127.3.245.62:33005)
0620 07:11:48.712885 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:48.712889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:48.713843 (+ 954us) server_negotiation.cc:407] Connection header received
0620 07:11:48.715267 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:48.715271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:48.715332 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:48.715438 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:48.716940 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.717593 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.719000 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.719270 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.719365 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.721825 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:48.721828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:48.721830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:48.721855 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:48.723456 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.724090 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.724093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.724094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.724176 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.724579 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.724581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.724582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.725059 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:48.725189 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:48.725728 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 07:11:48.725857 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:11:48.726971 7570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.713194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:33005 (local address 127.3.245.3:55325)
0620 07:11:48.713690 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:48.713735 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:48.715024 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:48.715617 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:48.715624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:48.715984 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:48.716755 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.716775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.717736 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.717743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.718887 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:48.718899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.719417 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.719420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:48.719577 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.720157 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:48.720174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:48.721660 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:48.723569 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.723575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.723590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.723985 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.724285 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:48.724288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:48.724289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:48.724434 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:48.725197 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:48.725207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:48.725482 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 07:11:48.725669 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 07:11:48.726132 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
I20260620 07:11:48.728193 7568 heartbeater.cc:347] Connected to a master server at 127.3.245.62:33005
I20260620 07:11:48.728541 7568 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:48.729112 7568 heartbeater.cc:511] Master 127.3.245.62:33005 requested a full tablet report, sending...
I20260620 07:11:48.730187 7109 ts_manager.cc:194] Registered new tserver with Master: 241b9ef96f83493b90ca946c45620143 (127.3.245.3:36613)
I20260620 07:11:48.730904 7109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:55325
I20260620 07:11:48.731129 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939508, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:48.757126 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:48.748484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.3.245.62:33005)
0620 07:11:48.748652 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:48.748659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:48.748678 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:11:48.748735 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:48.748737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:48.748777 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:48.748859 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:48.749582 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.750116 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.751037 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:48.751258 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:48.751361 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:48.754061 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:48.754069 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:48.754072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:48.754098 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:48.755365 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.755810 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.755814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.755816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.755897 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:48.756163 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:48.756167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:48.756169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:48.756396 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:48.756543 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:48.756824 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:11:48.756954 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 07:11:48.757501 7109 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42226
I20260620 07:11:48.758314 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7171
I20260620 07:11:48.764233 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7305
I20260620 07:11:48.769873 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7439
I20260620 07:11:48.775453 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7077
2026-06-20T07:11:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2186 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7580](info): set up 2 sockets
Jun 20 07:11:48 dist-test-slave-ldfr krb5kdc[7580](info): commencing operation
krb5kdc: starting...
W20260620 07:11:49.863286 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 07:11:50.076527 4052 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.213s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 07:11:50.276723 4052 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.200s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 07:11:50.597500 4052 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.321s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/test-user.keytab.
W20260620 07:11:51.313472 4052 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.716s user 0.001s sys 0.001s
Jun 20 07:11:51 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
W20260620 07:11:52.593246 4052 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.245.62: real 1.257s user 0.000s sys 0.003s
I20260620 07:11:52.593986 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44603
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:44603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:52.713152 7593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:52.713521 7593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:52.713613 7593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:52.717957 7593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:52.718030 7593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:52.718051 7593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:52.718070 7593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:52.718086 7593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:52.724213 7593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:44603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:52.725414 7593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:52.726505 7593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:52.735642 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:52.735847 7593 server_base.cc:1034] running on GCE node
W20260620 07:11:52.735728 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:52.735638 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:52.736541 7593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:52.737782 7593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:52.739022 7593 hybrid_clock.cc:648] HybridClock initialized: now 1781939512738991 us; error 51 us; skew 500 ppm
Jun 20 07:11:52 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:52.747378 7593 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:52.748912 7593 webserver.cc:460] Webserver started at http://127.3.245.62:40705/ using document root <none> and password file <none>
I20260620 07:11:52.749533 7593 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:52.749600 7593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:52.749773 7593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:52.752002 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "5d2b396b91c64653bdf25c9b82ad63cc"
format_stamp: "Formatted at 2026-06-20 07:11:52 on dist-test-slave-ldfr"
server_key: "f583156da5edefdbc81ca8ec280b1e46"
server_key_iv: "750d3650085ed5d7c902e9238584f08c"
server_key_version: "encryptionkey@0"
I20260620 07:11:52.752590 7593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "5d2b396b91c64653bdf25c9b82ad63cc"
format_stamp: "Formatted at 2026-06-20 07:11:52 on dist-test-slave-ldfr"
server_key: "f583156da5edefdbc81ca8ec280b1e46"
server_key_iv: "750d3650085ed5d7c902e9238584f08c"
server_key_version: "encryptionkey@0"
I20260620 07:11:52.757009 7593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:52.759706 7608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:52.761185 7593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:52.761293 7593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "5d2b396b91c64653bdf25c9b82ad63cc"
format_stamp: "Formatted at 2026-06-20 07:11:52 on dist-test-slave-ldfr"
server_key: "f583156da5edefdbc81ca8ec280b1e46"
server_key_iv: "750d3650085ed5d7c902e9238584f08c"
server_key_version: "encryptionkey@0"
I20260620 07:11:52.761387 7593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:52.784324 7593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:52.787717 7593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:52.787931 7593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:52.795861 7593 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:44603
I20260620 07:11:52.795884 7660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:44603 every 8 connection(s)
I20260620 07:11:52.796994 7593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:52.799962 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:52.801303 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7593
I20260620 07:11:52.801409 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:52.801709 4052 external_mini_cluster.cc:1428] Setting key dfa93f478fc7c5f1e23682c60221346c
I20260620 07:11:52.806244 7661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: Bootstrap starting.
I20260620 07:11:52.808579 7661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:52.809435 7661 log.cc:824] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:11:52 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:52.811576 7661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: No bootstrap required, opened a new log
I20260620 07:11:52.815336 7661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } }
I20260620 07:11:52.815594 7661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:52.815670 7661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d2b396b91c64653bdf25c9b82ad63cc, State: Initialized, Role: FOLLOWER
I20260620 07:11:52.816174 7661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } }
I20260620 07:11:52.816332 7661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:52.816428 7661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:52.816550 7661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:52.816972 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:52.803188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.3.245.62:44603)
0620 07:11:52.803774 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:52.803789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:52.803825 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:11:52.804489 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:52.804506 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:52.804958 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:52.805302 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:52.806140 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:52.807198 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:52.808052 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:52.808561 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:52.808770 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:52.812022 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:52.812027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:52.812041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:52.812068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:52.813828 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:52.814290 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:52.814296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:52.814301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:52.814407 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:52.814770 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:52.814774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:52.814777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:52.815191 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:52.815392 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:52.815672 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:11:52.816003 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":124,"threads_started":1}
I20260620 07:11:52.888010 7661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } }
I20260620 07:11:52.888661 7661 leader_election.cc:302] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d2b396b91c64653bdf25c9b82ad63cc; no voters:
I20260620 07:11:52.889144 7661 leader_election.cc:288] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:52.889328 7666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:52.889653 7666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [term 1 LEADER]: Becoming Leader. State: Replica: 5d2b396b91c64653bdf25c9b82ad63cc, State: Running, Role: LEADER
I20260620 07:11:52.890027 7666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } }
I20260620 07:11:52.890765 7661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:52.891988 7667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } } }
I20260620 07:11:52.892148 7667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:52.892537 7668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d2b396b91c64653bdf25c9b82ad63cc. Latest consensus state: current_term: 1 leader_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d2b396b91c64653bdf25c9b82ad63cc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44603 } } }
I20260620 07:11:52.892733 7668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc [sys.catalog]: This master's current role is: LEADER
W20260620 07:11:52.896111 7681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:52.896250 7681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:52.896502 7675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:52.897251 7675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:52.903683 7675 catalog_manager.cc:1347] Generated new cluster ID: 9e5f79c4845e49a580638a1b7859f33b
I20260620 07:11:52.903775 7675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:52.942987 7675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:52.944267 7675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:52.950438 7675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d2b396b91c64653bdf25c9b82ad63cc: Generated new TSK 0
I20260620 07:11:52.951206 7675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
W20260620 07:11:53.175624 4052 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.245.1: real 0.215s user 0.000s sys 0.003s
I20260620 07:11:53.176281 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:53.293978 7687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:53.294279 7687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:53.294350 7687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:53.298482 7687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:53.298584 7687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:53.298676 7687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:53.304162 7687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:53.305303 7687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:53.306406 7687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:53.316025 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:53.316227 7687 server_base.cc:1034] running on GCE node
W20260620 07:11:53.316111 7695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:53.316043 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:53.316859 7687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:53.317520 7687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:53.318681 7687 hybrid_clock.cc:648] HybridClock initialized: now 1781939513318647 us; error 46 us; skew 500 ppm
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:53.321631 7687 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:53.322528 7687 webserver.cc:460] Webserver started at http://127.3.245.1:45359/ using document root <none> and password file <none>
I20260620 07:11:53.323114 7687 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:53.323176 7687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:53.323380 7687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:53.325498 7687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "5319789653274e16a462967ace9815e3"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "6e07802e6c799ce761bbdca6b7cdf62b"
server_key_iv: "3b4015e50216eb2a698801d4166356bf"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.326092 7687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "5319789653274e16a462967ace9815e3"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "6e07802e6c799ce761bbdca6b7cdf62b"
server_key_iv: "3b4015e50216eb2a698801d4166356bf"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.330053 7687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:11:53.332700 7702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.334033 7687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:53.334169 7687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "5319789653274e16a462967ace9815e3"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "6e07802e6c799ce761bbdca6b7cdf62b"
server_key_iv: "3b4015e50216eb2a698801d4166356bf"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.334283 7687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:53.355422 7687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:53.359150 7687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:53.359426 7687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:53.360070 7687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:53.360980 7687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:53.361027 7687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.361095 7687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:53.361125 7687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.370234 7687 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37561
I20260620 07:11:53.370256 7815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37561 every 8 connection(s)
I20260620 07:11:53.371150 7687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:53.374008 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7687
I20260620 07:11:53.374127 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:53.374423 4052 external_mini_cluster.cc:1428] Setting key 442daa044653b6cd4b91f68c9de7dc01
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:11:53.386574 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.373167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:44865 (local address 127.3.245.62:44603)
0620 07:11:53.373434 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:53.373439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:53.374265 (+ 826us) server_negotiation.cc:407] Connection header received
0620 07:11:53.375957 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:53.375961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:53.376014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:53.376109 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:53.377367 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.378001 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.379327 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.379585 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.379674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.382889 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:53.382893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:53.382895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:53.382920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:53.384456 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.385023 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.385031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.385033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.385116 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.385455 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.385458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.385460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.385645 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:53.385747 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:53.386252 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 07:11:53.386383 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:11:53.387532 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.373645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44603 (local address 127.3.245.1:44865)
0620 07:11:53.374111 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:53.374164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:53.375719 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:53.376262 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:53.376269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:53.376649 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:53.377149 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.377163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.378149 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.378157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.379202 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.379212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.379744 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.379751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.379992 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.380856 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:53.380885 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:53.382700 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:53.384591 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.384597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.384649 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.384913 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.385205 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.385209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.385211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.385359 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.385745 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:53.385752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:53.386015 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 07:11:53.386199 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 07:11:53.386540 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":133,"threads_started":1}
I20260620 07:11:53.388824 7816 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44603
I20260620 07:11:53.389079 7816 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:53.389576 7816 heartbeater.cc:511] Master 127.3.245.62:44603 requested a full tablet report, sending...
I20260620 07:11:53.391326 7625 ts_manager.cc:194] Registered new tserver with Master: 5319789653274e16a462967ace9815e3 (127.3.245.1:37561)
I20260620 07:11:53.393105 7625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:44865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:53.403695 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:53.518959 7822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:53.519215 7822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:53.519260 7822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:53.523401 7822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:53.523470 7822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:53.523532 7822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:53.529147 7822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:53.530169 7822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:53.531134 7822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:53.541133 7828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:53.541205 7822 server_base.cc:1034] running on GCE node
W20260620 07:11:53.541163 7830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:53.541425 7827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:53.541879 7822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:53.542582 7822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:53.543772 7822 hybrid_clock.cc:648] HybridClock initialized: now 1781939513543734 us; error 48 us; skew 500 ppm
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:53.547191 7822 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:53.548182 7822 webserver.cc:460] Webserver started at http://127.3.245.2:35547/ using document root <none> and password file <none>
I20260620 07:11:53.548837 7822 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:53.548902 7822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:53.549110 7822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:53.551172 7822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "c6c71865662e4bdfa11378be7cb1e04f"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "2c72b8252bed2a0c6d49025b90a661ac"
server_key_iv: "312f92ef3703bf1323ce007b26a7011a"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.551735 7822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "c6c71865662e4bdfa11378be7cb1e04f"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "2c72b8252bed2a0c6d49025b90a661ac"
server_key_iv: "312f92ef3703bf1323ce007b26a7011a"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.555740 7822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:11:53.558414 7837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.559888 7822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:53.560038 7822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "c6c71865662e4bdfa11378be7cb1e04f"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "2c72b8252bed2a0c6d49025b90a661ac"
server_key_iv: "312f92ef3703bf1323ce007b26a7011a"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.560160 7822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:53.586102 7822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:53.589809 7822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:53.590020 7822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:53.590641 7822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:53.591487 7822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:53.591552 7822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.591621 7822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:53.591665 7822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.600749 7822 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:37831
I20260620 07:11:53.600862 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:37831 every 8 connection(s)
I20260620 07:11:53.601689 7822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:53.610464 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7822
I20260620 07:11:53.610575 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:53.610854 4052 external_mini_cluster.cc:1428] Setting key 0658920f01c7002647632871ba8c4b86
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:53.617206 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.603576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:47401 (local address 127.3.245.62:44603)
0620 07:11:53.603724 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:53.603728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:53.604645 (+ 917us) server_negotiation.cc:407] Connection header received
0620 07:11:53.606210 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:53.606214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:53.606277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:53.606370 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:53.607738 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.608340 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.609713 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.609980 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.610104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.613019 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:53.613026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:53.613031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:53.613067 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:53.614657 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.615372 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.615376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.615377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.615456 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.615814 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.615820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.615825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.616110 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:53.616211 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:53.616792 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 07:11:53.616958 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:11:53.617954 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.603911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44603 (local address 127.3.245.2:47401)
0620 07:11:53.604472 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:53.604518 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:53.605987 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:53.606554 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:53.606561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:53.607041 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:53.607578 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.607593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.608495 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.608499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.609573 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.609583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.610173 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.610180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.610360 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.611087 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:53.611105 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:53.612842 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:53.614777 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.614786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.614805 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.615209 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.615538 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.615541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.615542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.615702 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.616219 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:53.616224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:53.616502 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 07:11:53.616704 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 07:11:53.617029 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260620 07:11:53.619129 7951 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44603
I20260620 07:11:53.619360 7951 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:53.619916 7951 heartbeater.cc:511] Master 127.3.245.62:44603 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:11:53.620996 7625 ts_manager.cc:194] Registered new tserver with Master: c6c71865662e4bdfa11378be7cb1e04f (127.3.245.2:37831)
I20260620 07:11:53.621558 7625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:47401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:53.639916 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:53.750830 7956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:53.751137 7956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:53.751219 7956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:53.755331 7956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:53.755415 7956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:53.755501 7956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:53.760972 7956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:53.762099 7956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:53.763131 7956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:53.773191 7964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:53.773273 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:53.773188 7962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:53.773523 7956 server_base.cc:1034] running on GCE node
I20260620 07:11:53.773906 7956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:53.774541 7956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:53.775764 7956 hybrid_clock.cc:648] HybridClock initialized: now 1781939513775735 us; error 42 us; skew 500 ppm
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:53.778820 7956 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:53.779721 7956 webserver.cc:460] Webserver started at http://127.3.245.3:33535/ using document root <none> and password file <none>
I20260620 07:11:53.780272 7956 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:53.780341 7956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:53.780517 7956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:53.782848 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "663f02b59e214afe97f9cf5b17df61f7"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "f29c4376f2957c471b678b0106799146"
server_key_iv: "c98f97e6fd548a3091c87f721d373bf8"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.783408 7956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "663f02b59e214afe97f9cf5b17df61f7"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "f29c4376f2957c471b678b0106799146"
server_key_iv: "c98f97e6fd548a3091c87f721d373bf8"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.787460 7956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:53.790273 7971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.791636 7956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:53.791769 7956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "663f02b59e214afe97f9cf5b17df61f7"
format_stamp: "Formatted at 2026-06-20 07:11:53 on dist-test-slave-ldfr"
server_key: "f29c4376f2957c471b678b0106799146"
server_key_iv: "c98f97e6fd548a3091c87f721d373bf8"
server_key_version: "encryptionkey@0"
I20260620 07:11:53.791904 7956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:53.807121 7956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:53.810212 7956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:53.810422 7956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:53.811033 7956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:53.811846 7956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:53.811901 7956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.811940 7956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:53.811990 7956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:53.821554 7956 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:42897
I20260620 07:11:53.821595 8084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:42897 every 8 connection(s)
I20260620 07:11:53.822769 7956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:53.826634 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 7956
I20260620 07:11:53.826812 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:53.827190 4052 external_mini_cluster.cc:1428] Setting key d8b6695cd8bf566d314da12b2c53bb6c
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:53.838791 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.824945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:48623 (local address 127.3.245.62:44603)
0620 07:11:53.825103 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:53.825108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:53.826077 (+ 969us) server_negotiation.cc:407] Connection header received
0620 07:11:53.827976 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:53.827980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:53.828044 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:53.828150 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:53.829596 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.830186 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.831444 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.831699 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.831817 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.834494 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:53.834497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:53.834499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:53.834524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:53.836408 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.837056 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.837059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.837061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.837140 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.837476 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.837478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.837479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.837679 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:53.837758 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:53.838432 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 07:11:53.838596 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:11:53.839692 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.825417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44603 (local address 127.3.245.3:48623)
0620 07:11:53.825920 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:53.825966 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:53.827729 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:53.828334 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:53.828342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:53.828867 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:53.829451 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.829468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.830319 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.830329 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.831306 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:53.831316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.831866 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.831871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:53.832090 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.832741 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:53.832759 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:53.834322 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:53.836534 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.836539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.836552 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.836955 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.837232 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:53.837234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:53.837236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:53.837387 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:53.837803 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:53.837838 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:53.838095 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 07:11:53.838366 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 07:11:53.838745 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260620 07:11:53.840965 8085 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44603
I20260620 07:11:53.841236 8085 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:53.841786 8085 heartbeater.cc:511] Master 127.3.245.62:44603 requested a full tablet report, sending...
I20260620 07:11:53.842737 7624 ts_manager.cc:194] Registered new tserver with Master: 663f02b59e214afe97f9cf5b17df61f7 (127.3.245.3:42897)
I20260620 07:11:53.843282 7624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:48623
I20260620 07:11:53.853797 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:11:53.880725 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.870413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.3.245.1:37561)
0620 07:11:53.870942 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:53.870965 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:53.870987 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:11:53.871039 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:53.871045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:53.871230 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:53.871393 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:53.872192 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.873053 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.873929 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.874314 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.874445 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.877095 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:53.877104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:53.877119 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:53.877146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:53.878814 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.879300 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.879305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.879306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.879384 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.879658 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.879662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.879664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.879920 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:53.880152 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:53.880331 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:11:53.880482 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":124,"threads_started":1}
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[7580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:53.891911 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:53.883503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.3.245.62:44603)
0620 07:11:53.883651 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:53.883656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:53.883745 (+ 89us) server_negotiation.cc:407] Connection header received
0620 07:11:53.883916 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:53.883919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:53.883961 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:53.884045 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:53.884737 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.885263 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.886038 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:53.886257 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:53.886362 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:53.888708 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:53.888712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:53.888720 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:53.888744 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:53.890244 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.890649 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.890653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.890655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.890737 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:53.890994 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:53.890999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:53.891000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:53.891222 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:53.891361 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:53.891621 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 07:11:53.891740 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 07:11:53.892316 7624 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50388
I20260620 07:11:53.892966 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7687
I20260620 07:11:53.899310 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7822
I20260620 07:11:53.904832 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7956
I20260620 07:11:53.909888 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 7593
2026-06-20T07:11:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5134 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[8097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[8097](info): set up 2 sockets
Jun 20 07:11:53 dist-test-slave-ldfr krb5kdc[8097](info): commencing operation
krb5kdc: starting...
W20260620 07:11:54.965523 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:11:55.092738 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:32991
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:55.214061 8110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:55.214425 8110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:55.214514 8110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:55.219373 8110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:55.219458 8110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:55.219484 8110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:55.219529 8110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:55.219573 8110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:55.225894 8110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:55.227128 8110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:55.228293 8110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:55.237526 8116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.237519 8115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.237536 8118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:55.237967 8110 server_base.cc:1034] running on GCE node
I20260620 07:11:55.238531 8110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:55.239679 8110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:55.240897 8110 hybrid_clock.cc:648] HybridClock initialized: now 1781939515240875 us; error 35 us; skew 500 ppm
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:55.244066 8110 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:11:55.244962 8110 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:55.245064 8110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:55.245322 8110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:55.247611 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "adb2bdde74114786ad5c7a6d9e86e3f1"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "852f5c9c3b9aa669448e1b672554d6d0"
server_key_iv: "b9b6d065a78ca3a179d35f72467db98f"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.248255 8110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "adb2bdde74114786ad5c7a6d9e86e3f1"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "852f5c9c3b9aa669448e1b672554d6d0"
server_key_iv: "b9b6d065a78ca3a179d35f72467db98f"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.252388 8110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 07:11:55.255134 8124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.256570 8110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:55.256790 8110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "adb2bdde74114786ad5c7a6d9e86e3f1"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "852f5c9c3b9aa669448e1b672554d6d0"
server_key_iv: "b9b6d065a78ca3a179d35f72467db98f"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.256928 8110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:55.279847 8110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:55.283376 8110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:55.283600 8110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:55.290845 8110 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:32991
I20260620 07:11:55.291179 8176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:32991 every 8 connection(s)
I20260620 07:11:55.291554 8110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:55.294610 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:55.299721 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8110
I20260620 07:11:55.299923 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:55.300202 4052 external_mini_cluster.cc:1428] Setting key af0576b611b08c436ea4314d0f7efcfa
I20260620 07:11:55.300453 8177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: Bootstrap starting.
I20260620 07:11:55.303020 8177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:55.303858 8177 log.cc:824] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:55.305969 8177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: No bootstrap required, opened a new log
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:55.310545 8177 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } }
I20260620 07:11:55.310808 8177 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:55.310870 8177 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb2bdde74114786ad5c7a6d9e86e3f1, State: Initialized, Role: FOLLOWER
I20260620 07:11:55.311405 8177 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } }
I20260620 07:11:55.311604 8177 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:55.311679 8177 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:55.311795 8177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:55.313045 8177 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } }
I20260620 07:11:55.313457 8177 leader_election.cc:302] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb2bdde74114786ad5c7a6d9e86e3f1; no voters:
I20260620 07:11:55.313822 8177 leader_election.cc:288] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:55.313982 8182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:55.314237 8182 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [term 1 LEADER]: Becoming Leader. State: Replica: adb2bdde74114786ad5c7a6d9e86e3f1, State: Running, Role: LEADER
I20260620 07:11:55.314603 8182 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } }
I20260620 07:11:55.315232 8177 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:55.316350 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb2bdde74114786ad5c7a6d9e86e3f1. Latest consensus state: current_term: 1 leader_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } } }
I20260620 07:11:55.316491 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:55.316715 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:55.301794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.3.245.62:32991)
0620 07:11:55.302300 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:55.302324 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:55.302363 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:11:55.303133 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:55.303165 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:55.303562 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:55.303881 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:55.304788 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.305857 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.306989 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.307488 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.307617 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:55.310512 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:55.310517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:55.310531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:55.310558 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:55.312701 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.313123 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.313129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.313135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.313281 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.313599 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.313602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.313604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.314192 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:55.315060 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:55.315233 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:11:55.315602 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":157,"threads_started":1}
I20260620 07:11:55.316743 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb2bdde74114786ad5c7a6d9e86e3f1" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32991 } } }
I20260620 07:11:55.316825 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1 [sys.catalog]: This master's current role is: LEADER
W20260620 07:11:55.320427 8197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:55.320554 8197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:55.320667 8191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:55.321537 8191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:55.327087 8191 catalog_manager.cc:1347] Generated new cluster ID: 68e164614d8f43e99b1ca42f707d2d33
I20260620 07:11:55.327165 8191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:55.335870 8191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:55.336894 8191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:55.346637 8191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: Generated new TSK 0
I20260620 07:11:55.347383 8191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:11:55.382181 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:55.501441 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:55.501711 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:55.501755 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:55.505780 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:55.505846 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:55.505908 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:55.511317 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:55.512372 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:55.513438 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:55.523159 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.523216 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.523236 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:55.523469 8203 server_base.cc:1034] running on GCE node
I20260620 07:11:55.523898 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:55.524573 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:55.525776 8203 hybrid_clock.cc:648] HybridClock initialized: now 1781939515525740 us; error 54 us; skew 500 ppm
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:55.528728 8203 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:11:55.529516 8203 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:55.529580 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:55.529747 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:55.531765 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "fd71e33808a94d9c87fed9606230e142"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "d1ed026e452b6ef7251c6df920d580c8"
server_key_iv: "f00ff2d55ea4d8d12bc78b33b698c6a9"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.532280 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "fd71e33808a94d9c87fed9606230e142"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "d1ed026e452b6ef7251c6df920d580c8"
server_key_iv: "f00ff2d55ea4d8d12bc78b33b698c6a9"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.536312 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 07:11:55.538947 8217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.540166 8203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:11:55.540295 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "fd71e33808a94d9c87fed9606230e142"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "d1ed026e452b6ef7251c6df920d580c8"
server_key_iv: "f00ff2d55ea4d8d12bc78b33b698c6a9"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.540406 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:55.566952 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:55.570464 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:55.570772 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:55.571372 8203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:55.572216 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:55.572263 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.572327 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:55.572357 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.581018 8203 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:40591
I20260620 07:11:55.581089 8330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:40591 every 8 connection(s)
I20260620 07:11:55.581645 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:55.589762 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8203
I20260620 07:11:55.589907 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:55.590186 4052 external_mini_cluster.cc:1428] Setting key fbc728446f0144dd0f3647d30affaae2
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:55.597168 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:55.583433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:35547 (local address 127.3.245.62:32991)
0620 07:11:55.583610 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:55.583614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:55.584518 (+ 904us) server_negotiation.cc:407] Connection header received
0620 07:11:55.586021 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:55.586024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:55.586071 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:55.586137 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:55.587482 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.588072 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.589485 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.589744 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.589881 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:55.592732 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:55.592736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:55.592738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:55.592765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:55.594637 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.595198 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.595206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.595211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.595330 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.595694 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.595701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.595706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.595994 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:55.596118 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:55.596726 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 07:11:55.596890 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:11:55.597783 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:55.583832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:32991 (local address 127.3.245.1:35547)
0620 07:11:55.584306 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:55.584352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:55.585782 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:55.586328 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:55.586335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:55.586764 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:55.587296 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:55.587310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.588228 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.588231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:55.589345 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:55.589356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.589949 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.589953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:55.590124 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:55.590757 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:55.590781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:55.592437 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:55.594791 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:55.594796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:55.594808 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:55.595089 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:55.595452 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:55.595454 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:55.595456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:55.595596 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:55.596103 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:55.596108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:55.596412 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 07:11:55.596649 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 07:11:55.596922 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260620 07:11:55.599059 8331 heartbeater.cc:347] Connected to a master server at 127.3.245.62:32991
I20260620 07:11:55.599349 8331 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:55.600067 8331 heartbeater.cc:511] Master 127.3.245.62:32991 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:11:55.601699 8141 ts_manager.cc:194] Registered new tserver with Master: fd71e33808a94d9c87fed9606230e142 (127.3.245.1:40591)
I20260620 07:11:55.602833 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:35547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:11:55.619788 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:55.729759 8336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:55.730094 8336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:55.730167 8336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:55.734488 8336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:55.734584 8336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:55.734679 8336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:55.740101 8336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:55.741278 8336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:55.742342 8336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:55.752343 8344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:55.752381 8336 server_base.cc:1034] running on GCE node
W20260620 07:11:55.752343 8342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.752343 8341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:55.753046 8336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:55.753695 8336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:55.754864 8336 hybrid_clock.cc:648] HybridClock initialized: now 1781939515754843 us; error 32 us; skew 500 ppm
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:55.757747 8336 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:11:55.758559 8336 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:55.758647 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:55.758875 8336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:55.761133 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "ad73b773df8249208383d805453b1366"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "13e3341e95a09f0dc297259ef36cd728"
server_key_iv: "8b2ce21fff29aad0e3ae252ca96e0383"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.761745 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "ad73b773df8249208383d805453b1366"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "13e3341e95a09f0dc297259ef36cd728"
server_key_iv: "8b2ce21fff29aad0e3ae252ca96e0383"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.765709 8336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:11:55.768254 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.769630 8336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:11:55.769764 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "ad73b773df8249208383d805453b1366"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "13e3341e95a09f0dc297259ef36cd728"
server_key_iv: "8b2ce21fff29aad0e3ae252ca96e0383"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.769883 8336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:55.780452 8336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:55.783970 8336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:55.784181 8336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:55.784816 8336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:55.785660 8336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:55.785727 8336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.785811 8336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:55.785858 8336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.794754 8336 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:33427
I20260620 07:11:55.794770 8463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:33427 every 8 connection(s)
I20260620 07:11:55.795367 8336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:55.796010 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8336
I20260620 07:11:55.796133 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:55.796429 4052 external_mini_cluster.cc:1428] Setting key 39c91e34bf8ab527e8bd0fb4d946fd02
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:11:55.811342 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:55.797301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:53309 (local address 127.3.245.62:32991)
0620 07:11:55.797531 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:55.797535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:55.798409 (+ 874us) server_negotiation.cc:407] Connection header received
0620 07:11:55.800008 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:55.800011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:55.800056 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:55.800125 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:55.801445 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.801995 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.803840 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.804063 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.804148 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:55.807290 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:55.807293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:55.807295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:55.807319 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:55.809004 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.809569 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.809573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.809575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.809658 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:55.810045 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:55.810051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:55.810056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:55.810327 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:55.810439 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:55.811082 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 07:11:55.811216 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 07:11:55.812636 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:55.797742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:32991 (local address 127.3.245.2:53309)
0620 07:11:55.798240 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:55.798293 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:55.799774 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:55.800310 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:55.800331 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:55.800807 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:55.801280 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:55.801293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.802125 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.802132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:55.803684 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:55.803700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:55.804227 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:55.804233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:55.804485 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:55.805231 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:55.805252 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:55.807088 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:55.809132 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:55.809139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:55.809156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:55.809459 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:55.809776 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:55.809778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:55.809780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:55.809948 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:55.810499 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:55.810506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:55.810764 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 07:11:55.811025 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 07:11:55.811458 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":126,"threads_started":1}
I20260620 07:11:55.813994 8464 heartbeater.cc:347] Connected to a master server at 127.3.245.62:32991
I20260620 07:11:55.814284 8464 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:55.814765 8464 heartbeater.cc:511] Master 127.3.245.62:32991 requested a full tablet report, sending...
I20260620 07:11:55.815862 8141 ts_manager.cc:194] Registered new tserver with Master: ad73b773df8249208383d805453b1366 (127.3.245.2:33427)
I20260620 07:11:55.816501 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:53309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:11:55.828040 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:11:55.944777 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:55.945098 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:55.945173 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:55.949514 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:55.949585 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:55.949689 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:55.955603 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:55.956856 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:55.957923 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:55.967931 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.967909 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:55.967911 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:55.968328 8469 server_base.cc:1034] running on GCE node
I20260620 07:11:55.968974 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:55.969771 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:55.971014 8469 hybrid_clock.cc:648] HybridClock initialized: now 1781939515970996 us; error 39 us; skew 500 ppm
Jun 20 07:11:55 dist-test-slave-ldfr krb5kdc[8097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:11:55.974294 8469 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:11:55.975123 8469 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:55.975216 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:55.975505 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:55.977969 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "d04775d4b10148dc9292fe233f4ae37d"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "a5ef53db33a31e0f7988cfc0336cb10a"
server_key_iv: "804eb1ea5dca847a011ca27e232a03d4"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.978585 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "d04775d4b10148dc9292fe233f4ae37d"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "a5ef53db33a31e0f7988cfc0336cb10a"
server_key_iv: "804eb1ea5dca847a011ca27e232a03d4"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.982559 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:11:55.985368 8483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:55.986701 8469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:55.986855 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "d04775d4b10148dc9292fe233f4ae37d"
format_stamp: "Formatted at 2026-06-20 07:11:55 on dist-test-slave-ldfr"
server_key: "a5ef53db33a31e0f7988cfc0336cb10a"
server_key_iv: "804eb1ea5dca847a011ca27e232a03d4"
server_key_version: "encryptionkey@0"
I20260620 07:11:55.986995 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:56.001276 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:56.004704 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:56.004916 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:56.005525 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:56.006397 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:56.006469 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.006541 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:56.006582 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.016822 8469 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:46811
I20260620 07:11:56.016894 8596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:46811 every 8 connection(s)
I20260620 07:11:56.017479 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:56.024971 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8469
I20260620 07:11:56.025120 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:56.025434 4052 external_mini_cluster.cc:1428] Setting key 8fc579f11989342553a2e5ea19469b20
Jun 20 07:11:56 dist-test-slave-ldfr krb5kdc[8097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939515, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:11:56.032449 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.019390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:36777 (local address 127.3.245.62:32991)
0620 07:11:56.019549 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.019554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.020444 (+ 890us) server_negotiation.cc:407] Connection header received
0620 07:11:56.021752 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.021755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.021802 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.021907 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:56.023116 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.023666 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.025118 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.025417 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.025539 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.028595 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:56.028599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:56.028601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:56.028700 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:56.030074 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:56.030647 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:56.030650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:56.030652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:56.030738 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:56.031097 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:56.031100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:56.031102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:56.031293 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:56.031444 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.032140 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.032262 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:11:56.033226 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.019778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:32991 (local address 127.3.245.3:36777)
0620 07:11:56.020228 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.020275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.021552 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.022089 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.022096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.022503 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:56.022981 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.022994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.023807 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.023810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.024936 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.024950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.025609 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.025613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.025826 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.026508 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:11:56.026525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:56.028260 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:56.030210 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:56.030215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:56.030228 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:56.030535 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:56.030846 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:11:56.030849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:11:56.030850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:11:56.031001 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:11:56.031627 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:56.031634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:56.031881 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.032047 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.032339 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260620 07:11:56.034638 8597 heartbeater.cc:347] Connected to a master server at 127.3.245.62:32991
I20260620 07:11:56.034868 8597 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:56.035307 8597 heartbeater.cc:511] Master 127.3.245.62:32991 requested a full tablet report, sending...
I20260620 07:11:56.036458 8141 ts_manager.cc:194] Registered new tserver with Master: d04775d4b10148dc9292fe233f4ae37d (127.3.245.3:46811)
I20260620 07:11:56.037114 8141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:36777
I20260620 07:11:56.039788 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:56.050043 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.041812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.3.245.62:32991)
0620 07:11:56.041973 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.041977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.042077 (+ 100us) server_negotiation.cc:407] Connection header received
0620 07:11:56.042160 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.042162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.042201 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.042282 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:56.043066 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.043658 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.044663 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.044942 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.045059 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.046639 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:56.046643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:56.046646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:11:56.046672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:56.048129 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:56.048647 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:56.048651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:56.048653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:56.048744 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:11:56.049066 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:11:56.049070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:11:56.049072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:11:56.049311 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:56.049467 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.049720 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.049844 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:11:56.053121 8141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:56.056342 8141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:56.072412 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.3.245.2:33427)
0620 07:11:56.067641 (+ 1389us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.067656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.067686 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:11:56.067783 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.067791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.068016 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.068559 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:56.068788 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.069517 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.071025 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.071626 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.071803 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.071981 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.072137 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.072240 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":148,"threads_started":1}
I20260620 07:11:56.072409 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33427 (local address 127.0.0.1:59010)
0620 07:11:56.066819 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.066875 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.067028 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.068193 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.068199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.068252 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:56.068393 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.068405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.069666 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.069670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.070892 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.070902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.071825 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.071828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.071919 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.071952 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.072093 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.072224 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"spinlock_wait_cycles":11776,"thread_start_us":96,"threads_started":1}
I20260620 07:11:56.073443 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:46811 (local address 127.0.0.1:43116)
0620 07:11:56.067636 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.067653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.067768 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.068435 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.068448 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.068470 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:56.068558 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.068565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.070097 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.070100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.071577 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.071588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.073007 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.073010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.073101 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.073129 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.073209 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.073285 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":87,"thread_start_us":91,"threads_started":1}
I20260620 07:11:56.074044 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.3.245.3:46811)
0620 07:11:56.067411 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.067424 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.067869 (+ 445us) server_negotiation.cc:407] Connection header received
0620 07:11:56.067945 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.067950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.068114 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.068270 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:56.068721 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.069948 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.071751 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.072669 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.072778 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.072940 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.073723 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.073897 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":162,"threads_started":1}
I20260620 07:11:56.074409 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.3.245.1:40591)
0620 07:11:56.067427 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.067435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.067699 (+ 264us) server_negotiation.cc:407] Connection header received
0620 07:11:56.067833 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.067837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.068004 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.068172 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:56.068498 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.069878 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.072068 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.073153 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.073247 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.073393 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.074089 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.074198 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":114,"threads_started":1}
I20260620 07:11:56.073982 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.066297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:40591 (local address 127.0.0.1:54402)
0620 07:11:56.067427 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.067442 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.067581 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.068173 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.068179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.068202 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:56.068370 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.068377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.070077 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.070081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.071577 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.071588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.073472 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.073476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.073585 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.073606 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.073675 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.073743 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":95,"thread_start_us":86,"threads_started":1}
I20260620 07:11:56.076301 8398 tablet_service.cc:1467] Processing CreateTablet for tablet ec87eed671674c5187ebac6c35732bf3 (DEFAULT_TABLE table=test-table [id=b7e0881c941743bbaa68063073f4e5fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:56.076781 8531 tablet_service.cc:1467] Processing CreateTablet for tablet ec87eed671674c5187ebac6c35732bf3 (DEFAULT_TABLE table=test-table [id=b7e0881c941743bbaa68063073f4e5fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:56.077795 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec87eed671674c5187ebac6c35732bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:56.077805 8531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec87eed671674c5187ebac6c35732bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:56.077440 8265 tablet_service.cc:1467] Processing CreateTablet for tablet ec87eed671674c5187ebac6c35732bf3 (DEFAULT_TABLE table=test-table [id=b7e0881c941743bbaa68063073f4e5fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:56.078504 8265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec87eed671674c5187ebac6c35732bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:56.083974 8614 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Bootstrap starting.
I20260620 07:11:56.085700 8616 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: Bootstrap starting.
I20260620 07:11:56.086303 8614 tablet_bootstrap.cc:654] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:56.087440 8616 tablet_bootstrap.cc:654] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:56.087415 8614 log.cc:824] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:56.088447 8615 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Bootstrap starting.
I20260620 07:11:56.088732 8616 log.cc:824] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:56.089584 8614 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: No bootstrap required, opened a new log
I20260620 07:11:56.089805 8614 ts_tablet_manager.cc:1397] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:56.090564 8615 tablet_bootstrap.cc:654] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:56.091128 8616 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: No bootstrap required, opened a new log
I20260620 07:11:56.091472 8616 ts_tablet_manager.cc:1397] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 07:11:56.092078 8615 log.cc:824] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:56.093789 8615 tablet_bootstrap.cc:492] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: No bootstrap required, opened a new log
I20260620 07:11:56.093981 8615 ts_tablet_manager.cc:1397] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:11:56.093951 8614 raft_consensus.cc:348] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.094245 8614 raft_consensus.cc:374] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:56.094309 8614 raft_consensus.cc:729] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04775d4b10148dc9292fe233f4ae37d, State: Initialized, Role: FOLLOWER
I20260620 07:11:56.094789 8614 consensus_queue.cc:260] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.095363 8616 raft_consensus.cc:348] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.095602 8616 raft_consensus.cc:374] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:56.095650 8616 raft_consensus.cc:729] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd71e33808a94d9c87fed9606230e142, State: Initialized, Role: FOLLOWER
I20260620 07:11:56.095846 8614 ts_tablet_manager.cc:1428] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:11:56.096189 8597 heartbeater.cc:503] Master 127.3.245.62:32991 was elected leader, sending a full tablet report...
I20260620 07:11:56.096720 8616 consensus_queue.cc:260] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.097446 8331 heartbeater.cc:503] Master 127.3.245.62:32991 was elected leader, sending a full tablet report...
I20260620 07:11:56.097848 8616 ts_tablet_manager.cc:1428] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:11:56.099961 8615 raft_consensus.cc:348] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.100260 8615 raft_consensus.cc:374] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:56.100332 8615 raft_consensus.cc:729] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad73b773df8249208383d805453b1366, State: Initialized, Role: FOLLOWER
I20260620 07:11:56.100865 8615 consensus_queue.cc:260] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.101593 8464 heartbeater.cc:503] Master 127.3.245.62:32991 was elected leader, sending a full tablet report...
I20260620 07:11:56.101854 8615 ts_tablet_manager.cc:1428] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:11:56.227216 8621 raft_consensus.cc:482] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:56.227420 8621 raft_consensus.cc:504] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.228667 8621 leader_election.cc:288] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad73b773df8249208383d805453b1366 (127.3.245.2:33427), d04775d4b10148dc9292fe233f4ae37d (127.3.245.3:46811)
I20260620 07:11:56.232883 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.229032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:49149 (local address 127.3.245.3:46811)
0620 07:11:56.229221 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.229227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.229323 (+ 96us) server_negotiation.cc:407] Connection header received
0620 07:11:56.229581 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.229583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.229625 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.229713 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:56.230013 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.230488 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.231677 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.232218 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.232391 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.232458 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.232656 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.232739 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 07:11:56.232908 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.228945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:46811 (local address 127.3.245.1:49149)
0620 07:11:56.229219 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.229236 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.229454 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.229738 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.229741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.229762 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:56.229878 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.229883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.230598 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.230603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.231523 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.231532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.232391 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.232394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.232493 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.232516 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.232594 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.232696 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":26,"spinlock_wait_cycles":23936,"thread_start_us":105,"threads_started":1}
I20260620 07:11:56.233021 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.229058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48537 (local address 127.3.245.2:33427)
0620 07:11:56.229330 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.229334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.229403 (+ 69us) server_negotiation.cc:407] Connection header received
0620 07:11:56.229556 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.229561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.229620 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.229678 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:11:56.229984 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.230487 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.231983 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.232471 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.232588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.232671 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.232831 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.232907 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260620 07:11:56.233232 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.228954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:33427 (local address 127.3.245.1:48537)
0620 07:11:56.229310 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.229324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.229448 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.229716 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.229719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.229734 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:11:56.229866 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.229871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.230600 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.230604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.231828 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:56.231839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.232586 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.232592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:56.232708 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.232728 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.232800 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.232857 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":64,"spinlock_wait_cycles":8832,"thread_start_us":131,"threads_started":1}
I20260620 07:11:56.233752 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec87eed671674c5187ebac6c35732bf3" candidate_uuid: "fd71e33808a94d9c87fed9606230e142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04775d4b10148dc9292fe233f4ae37d" is_pre_election: true
I20260620 07:11:56.233875 8418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec87eed671674c5187ebac6c35732bf3" candidate_uuid: "fd71e33808a94d9c87fed9606230e142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad73b773df8249208383d805453b1366" is_pre_election: true
I20260620 07:11:56.234107 8551 raft_consensus.cc:2454] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71e33808a94d9c87fed9606230e142 in term 0.
I20260620 07:11:56.234170 8418 raft_consensus.cc:2454] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd71e33808a94d9c87fed9606230e142 in term 0.
I20260620 07:11:56.234724 8219 leader_election.cc:302] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d04775d4b10148dc9292fe233f4ae37d, fd71e33808a94d9c87fed9606230e142; no voters:
I20260620 07:11:56.235008 8621 raft_consensus.cc:2781] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:56.235105 8621 raft_consensus.cc:482] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:56.235162 8621 raft_consensus.cc:3037] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:56.236200 8621 raft_consensus.cc:504] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.236552 8621 leader_election.cc:288] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [CANDIDATE]: Term 1 election: Requested vote from peers ad73b773df8249208383d805453b1366 (127.3.245.2:33427), d04775d4b10148dc9292fe233f4ae37d (127.3.245.3:46811)
I20260620 07:11:56.236997 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec87eed671674c5187ebac6c35732bf3" candidate_uuid: "fd71e33808a94d9c87fed9606230e142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04775d4b10148dc9292fe233f4ae37d"
I20260620 07:11:56.236999 8418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec87eed671674c5187ebac6c35732bf3" candidate_uuid: "fd71e33808a94d9c87fed9606230e142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad73b773df8249208383d805453b1366"
I20260620 07:11:56.237107 8551 raft_consensus.cc:3037] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:56.237144 8418 raft_consensus.cc:3037] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:56.238195 8551 raft_consensus.cc:2454] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71e33808a94d9c87fed9606230e142 in term 1.
I20260620 07:11:56.238361 8418 raft_consensus.cc:2454] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd71e33808a94d9c87fed9606230e142 in term 1.
I20260620 07:11:56.238557 8219 leader_election.cc:302] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d04775d4b10148dc9292fe233f4ae37d, fd71e33808a94d9c87fed9606230e142; no voters:
I20260620 07:11:56.238744 8621 raft_consensus.cc:2781] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:56.238998 8621 raft_consensus.cc:686] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [term 1 LEADER]: Becoming Leader. State: Replica: fd71e33808a94d9c87fed9606230e142, State: Running, Role: LEADER
I20260620 07:11:56.239360 8621 consensus_queue.cc:237] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } }
I20260620 07:11:56.243255 8141 catalog_manager.cc:5495] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd71e33808a94d9c87fed9606230e142 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "fd71e33808a94d9c87fed9606230e142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd71e33808a94d9c87fed9606230e142" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:56.264163 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.260179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.3.245.1:40591)
0620 07:11:56.260343 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.260346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.260361 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:11:56.260453 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.260455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.260580 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.260716 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:11:56.261068 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.261686 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.262602 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:56.262841 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:56.262945 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:56.263282 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:11:56.263729 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:11:56.263829 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.263975 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.264038 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:11:56.268373 8598 tablet.cc:2369] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:56.270951 8551 raft_consensus.cc:1261] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 1 FOLLOWER]: Refusing update from remote peer fd71e33808a94d9c87fed9606230e142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:56.270951 8418 raft_consensus.cc:1261] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 1 FOLLOWER]: Refusing update from remote peer fd71e33808a94d9c87fed9606230e142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:56.271745 8625 consensus_queue.cc:1035] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad73b773df8249208383d805453b1366" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:56.271944 8621 consensus_queue.cc:1035] T ec87eed671674c5187ebac6c35732bf3 P fd71e33808a94d9c87fed9606230e142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04775d4b10148dc9292fe233f4ae37d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:56.281133 8629 mvcc.cc:205] Tried to move back new op lower bound from 7298824258637959168 to 7298824258519949312. Current Snapshot: MvccSnapshot[applied={T|T < 7298824258637959168}]
I20260620 07:11:56.282681 8630 mvcc.cc:205] Tried to move back new op lower bound from 7298824258637959168 to 7298824258519949312. Current Snapshot: MvccSnapshot[applied={T|T < 7298824258637959168}]
I20260620 07:11:56.287967 8140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:56.288182 8140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:56.291009 8140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adb2bdde74114786ad5c7a6d9e86e3f1: Sending DeleteTablet for 3 replicas of tablet ec87eed671674c5187ebac6c35732bf3
I20260620 07:11:56.292009 8265 tablet_service.cc:1514] Processing DeleteTablet for tablet ec87eed671674c5187ebac6c35732bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:56 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:54402
I20260620 07:11:56.292565 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8203
I20260620 07:11:56.293736 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet ec87eed671674c5187ebac6c35732bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:56 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:59010
I20260620 07:11:56.294323 8638 tablet_replica.cc:330] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: stopping tablet replica
I20260620 07:11:56.294399 8531 tablet_service.cc:1514] Processing DeleteTablet for tablet ec87eed671674c5187ebac6c35732bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:56 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:43116
I20260620 07:11:56.294745 8638 raft_consensus.cc:2229] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:56.295114 8638 raft_consensus.cc:2258] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:56.296649 8638 ts_tablet_manager.cc:1905] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:56.296667 8639 tablet_replica.cc:330] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: stopping tablet replica
I20260620 07:11:56.297087 8639 raft_consensus.cc:2229] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:56.297477 8639 raft_consensus.cc:2258] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:56.299131 8638 ts_tablet_manager.cc:1918] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:56.299247 8638 log.cc:1196] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/ec87eed671674c5187ebac6c35732bf3
I20260620 07:11:56.299535 8638 ts_tablet_manager.cc:1939] T ec87eed671674c5187ebac6c35732bf3 P ad73b773df8249208383d805453b1366: Deleting consensus metadata
I20260620 07:11:56.301009 8639 ts_tablet_manager.cc:1905] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:56.301278 8127 catalog_manager.cc:4841] TS ad73b773df8249208383d805453b1366 (127.3.245.2:33427): tablet ec87eed671674c5187ebac6c35732bf3 (table test-table [id=b7e0881c941743bbaa68063073f4e5fd]) successfully deleted
W20260620 07:11:56.303257 8128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:40591 (error 108)
I20260620 07:11:56.303767 8639 ts_tablet_manager.cc:1918] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:56.303879 8639 log.cc:1196] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/ec87eed671674c5187ebac6c35732bf3
I20260620 07:11:56.304105 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8336
I20260620 07:11:56.304224 8639 ts_tablet_manager.cc:1939] T ec87eed671674c5187ebac6c35732bf3 P d04775d4b10148dc9292fe233f4ae37d: Deleting consensus metadata
I20260620 07:11:56.305075 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.304219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:40591 (local address 127.0.0.1:54428)
0620 07:11:56.304921 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.304992 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608}
W20260620 07:11:56.305848 8128 catalog_manager.cc:4584] TS fd71e33808a94d9c87fed9606230e142 (127.3.245.1:40591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec87eed671674c5187ebac6c35732bf3: Network error: Client connection negotiation failed: client connection to 127.3.245.1:40591: connect: Connection refused (error 111)
I20260620 07:11:56.305958 8126 catalog_manager.cc:4841] TS d04775d4b10148dc9292fe233f4ae37d (127.3.245.3:46811): tablet ec87eed671674c5187ebac6c35732bf3 (table test-table [id=b7e0881c941743bbaa68063073f4e5fd]) successfully deleted
I20260620 07:11:56.310849 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8469
I20260620 07:11:56.319170 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8110
2026-06-20T07:11:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2410 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T07:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:56Z Disabled control of system clock
I20260620 07:11:56.351004 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35921
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:35921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:11:56.468221 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:56.468515 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:56.468554 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:56.472659 8644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:56.472723 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:56.472739 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:56.472756 8644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:56.472771 8644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:11:56.477777 8644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:35921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:56.478857 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:56.480238 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:56.489961 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:56.489964 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:56.490130 8644 server_base.cc:1034] running on GCE node
W20260620 07:11:56.489992 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:56.490899 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:56.492029 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:56.493204 8644 hybrid_clock.cc:648] HybridClock initialized: now 1781939516493185 us; error 34 us; skew 500 ppm
I20260620 07:11:56.495253 8644 webserver.cc:460] Webserver started at http://127.3.245.62:33143/ using document root <none> and password file <none>
I20260620 07:11:56.495905 8644 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:56.495976 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:56.496203 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:56.498541 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "c4cc9571861e4d32bb445d00deda3f42"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "c3323b124e45bfa570b5421fce8799f6"
server_key_iv: "634a7bcbf510405116c326afa44d0cb9"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.499155 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "c4cc9571861e4d32bb445d00deda3f42"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "c3323b124e45bfa570b5421fce8799f6"
server_key_iv: "634a7bcbf510405116c326afa44d0cb9"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.503507 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:56.506346 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.507769 8644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:11:56.507925 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "c4cc9571861e4d32bb445d00deda3f42"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "c3323b124e45bfa570b5421fce8799f6"
server_key_iv: "634a7bcbf510405116c326afa44d0cb9"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.508054 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:56.512984 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:56.513638 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:56.513826 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:56.520831 8644 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:35921
I20260620 07:11:56.520957 8710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:35921 every 8 connection(s)
I20260620 07:11:56.521829 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:56.525058 8711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:56.527665 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8644
I20260620 07:11:56.527797 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:56.528057 4052 external_mini_cluster.cc:1428] Setting key e9181138646f958f5a9f6835e4adb3dc
I20260620 07:11:56.531257 8711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: Bootstrap starting.
I20260620 07:11:56.533984 8711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:56.534135 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.529400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.3.245.62:35921)
0620 07:11:56.530085 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.530105 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.530149 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:11:56.530968 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.530991 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.531447 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.531805 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:56.532651 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:56.532657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:56.532673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:56.532686 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:56.532750 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:56.532773 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:56.532914 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.532981 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.533221 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260620 07:11:56.534785 8711 log.cc:824] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:56.536836 8711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: No bootstrap required, opened a new log
I20260620 07:11:56.540297 8711 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } }
I20260620 07:11:56.540558 8711 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:56.540657 8711 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cc9571861e4d32bb445d00deda3f42, State: Initialized, Role: FOLLOWER
I20260620 07:11:56.541092 8711 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } }
I20260620 07:11:56.541241 8711 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:56.541320 8711 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:56.541438 8711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:56.542385 8711 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } }
I20260620 07:11:56.542729 8711 leader_election.cc:302] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4cc9571861e4d32bb445d00deda3f42; no voters:
I20260620 07:11:56.543038 8711 leader_election.cc:288] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:56.543143 8716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:56.543411 8716 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [term 1 LEADER]: Becoming Leader. State: Replica: c4cc9571861e4d32bb445d00deda3f42, State: Running, Role: LEADER
I20260620 07:11:56.543722 8716 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } }
I20260620 07:11:56.544287 8711 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:56.545307 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4cc9571861e4d32bb445d00deda3f42. Latest consensus state: current_term: 1 leader_uuid: "c4cc9571861e4d32bb445d00deda3f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } } }
I20260620 07:11:56.545527 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:56.545711 8717 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4cc9571861e4d32bb445d00deda3f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4cc9571861e4d32bb445d00deda3f42" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35921 } } }
I20260620 07:11:56.545809 8717 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42 [sys.catalog]: This master's current role is: LEADER
W20260620 07:11:56.549260 8732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:56.549355 8732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:56.549458 8725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:56.550150 8725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:56.555568 8725 catalog_manager.cc:1347] Generated new cluster ID: 0c6f6d97a9504af0a132a3ae40792627
I20260620 07:11:56.555650 8725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:56.564180 8725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:56.564517 8725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:56.570977 8725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: Generated new TSK 0
I20260620 07:11:56.571653 8725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:11:56.578747 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--builtin_ntp_servers=127.3.245.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 07:11:56.693337 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:56.693615 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:56.693656 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:56.697885 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:56.697945 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:56.698033 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:11:56.702961 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:56.704159 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:56.705612 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:56.715452 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:56.715487 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:56.715516 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:56.715778 8735 server_base.cc:1034] running on GCE node
I20260620 07:11:56.716179 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:56.716877 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:56.718062 8735 hybrid_clock.cc:648] HybridClock initialized: now 1781939516718043 us; error 32 us; skew 500 ppm
I20260620 07:11:56.719872 8735 webserver.cc:460] Webserver started at http://127.3.245.1:35121/ using document root <none> and password file <none>
I20260620 07:11:56.720467 8735 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:56.720557 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:56.720810 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:56.723330 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "e9aec33acac24290b27ce077c02fff3c"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "457e55776c95fb813ad67a21dd9b940f"
server_key_iv: "fd914c25a33717d7fd9c0feaf3ed14c6"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.723909 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "e9aec33acac24290b27ce077c02fff3c"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "457e55776c95fb813ad67a21dd9b940f"
server_key_iv: "fd914c25a33717d7fd9c0feaf3ed14c6"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.727717 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:11:56.730427 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.731715 8735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:11:56.731859 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "e9aec33acac24290b27ce077c02fff3c"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "457e55776c95fb813ad67a21dd9b940f"
server_key_iv: "fd914c25a33717d7fd9c0feaf3ed14c6"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.731979 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:56.736876 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:56.737485 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:56.737665 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:56.738191 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:56.739081 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:56.739147 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.739214 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:56.739253 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.748613 8735 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:33301
I20260620 07:11:56.748651 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:33301 every 8 connection(s)
I20260620 07:11:56.749583 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:11:56.755383 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8735
I20260620 07:11:56.755537 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:11:56.755823 4052 external_mini_cluster.cc:1428] Setting key 6f547f5d46bfd1ab10fc500bf7b1be25
I20260620 07:11:56.756043 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.751476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:34843 (local address 127.3.245.62:35921)
0620 07:11:56.751612 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.751616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.752677 (+ 1061us) server_negotiation.cc:407] Connection header received
0620 07:11:56.754140 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.754143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.754193 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.754312 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:56.755505 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:56.755511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:56.755517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:56.755531 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:56.755556 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:56.755566 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:56.755666 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.755851 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.755899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:11:56.757005 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.751837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35921 (local address 127.3.245.1:34843)
0620 07:11:56.752512 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.752554 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.753911 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.754485 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.754493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.754588 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:56.755231 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:56.755233 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:56.755275 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:56.755351 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:56.755677 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:56.755688 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:56.755695 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.755896 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.756150 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260620 07:11:56.757872 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--builtin_ntp_servers=127.3.245.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:11:56.758320 8863 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35921
I20260620 07:11:56.758569 8863 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:56.759032 8863 heartbeater.cc:511] Master 127.3.245.62:35921 requested a full tablet report, sending...
I20260620 07:11:56.760900 8675 ts_manager.cc:194] Registered new tserver with Master: e9aec33acac24290b27ce077c02fff3c (127.3.245.1:33301)
W20260620 07:11:56.878968 8866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:56.879274 8866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:56.879317 8866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:56.883770 8866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:56.883925 8866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:56.884064 8866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:11:56.889533 8866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:56.890666 8866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:56.892064 8866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:56.904368 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:56.904523 8872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:56.904702 8866 server_base.cc:1034] running on GCE node
W20260620 07:11:56.904788 8871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:56.905385 8866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:56.906165 8866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:56.907349 8866 hybrid_clock.cc:648] HybridClock initialized: now 1781939516907312 us; error 54 us; skew 500 ppm
I20260620 07:11:56.909725 8866 webserver.cc:460] Webserver started at http://127.3.245.2:41749/ using document root <none> and password file <none>
I20260620 07:11:56.910415 8866 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:56.910486 8866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:56.910724 8866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:56.913421 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "407e429a98004d41b5406c23740c7e6a"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "e83914ed2983f375a8ec5baeae3a81d4"
server_key_iv: "fd5f112fd2e25ce592b8a633ceb69c40"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.914182 8866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "407e429a98004d41b5406c23740c7e6a"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "e83914ed2983f375a8ec5baeae3a81d4"
server_key_iv: "fd5f112fd2e25ce592b8a633ceb69c40"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.918228 8866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 07:11:56.921010 8880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.922363 8866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:56.922576 8866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "407e429a98004d41b5406c23740c7e6a"
format_stamp: "Formatted at 2026-06-20 07:11:56 on dist-test-slave-ldfr"
server_key: "e83914ed2983f375a8ec5baeae3a81d4"
server_key_iv: "fd5f112fd2e25ce592b8a633ceb69c40"
server_key_version: "encryptionkey@0"
I20260620 07:11:56.922727 8866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:56.927825 8866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:56.928478 8866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:56.928726 8866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:56.929301 8866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:56.930253 8866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:56.930337 8866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.930408 8866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:56.930461 8866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:56.939742 8866 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:34773
I20260620 07:11:56.939796 8993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:34773 every 8 connection(s)
I20260620 07:11:56.940765 8866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:11:56.945410 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8866
I20260620 07:11:56.945547 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:11:56.945886 4052 external_mini_cluster.cc:1428] Setting key c2133ec703a9d95f82c671848410abfe
I20260620 07:11:56.948236 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.942879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:54979 (local address 127.3.245.62:35921)
0620 07:11:56.943031 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:56.943036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:56.944412 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 07:11:56.946144 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:56.946148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:56.946202 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:56.946337 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:56.947750 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:56.947754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:56.947757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:56.947771 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:56.947795 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:56.947805 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:56.947901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:56.948043 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:11:56.948087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:11:56.948503 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--builtin_ntp_servers=127.3.245.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 07:11:56.949263 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:56.943347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35921 (local address 127.3.245.2:54979)
0620 07:11:56.944222 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:56.944273 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:56.945883 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:56.946557 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:56.946567 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:56.946691 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:56.947396 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:56.947400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:56.947459 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:56.947575 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:56.947898 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:56.947905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:56.947909 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:11:56.948050 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:11:56.948257 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260620 07:11:56.950443 8994 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35921
I20260620 07:11:56.950681 8994 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:56.951115 8994 heartbeater.cc:511] Master 127.3.245.62:35921 requested a full tablet report, sending...
I20260620 07:11:56.952180 8675 ts_manager.cc:194] Registered new tserver with Master: 407e429a98004d41b5406c23740c7e6a (127.3.245.2:34773)
W20260620 07:11:57.066020 8997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:57.066349 8997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:57.066424 8997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:57.071026 8997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:57.071131 8997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:57.071231 8997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:11:57.076464 8997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:57.077656 8997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:57.079082 8997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:57.088994 9002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:57.089003 9003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:57.088995 9005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:57.089162 8997 server_base.cc:1034] running on GCE node
I20260620 07:11:57.089861 8997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:57.090514 8997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:57.091709 8997 hybrid_clock.cc:648] HybridClock initialized: now 1781939517091669 us; error 60 us; skew 500 ppm
I20260620 07:11:57.093596 8997 webserver.cc:460] Webserver started at http://127.3.245.3:36031/ using document root <none> and password file <none>
I20260620 07:11:57.094179 8997 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:57.094271 8997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:57.094513 8997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:57.096561 8997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "26c6eb588e9146c19280f01845c768ff"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "b2728b542ed8144fe6e1e98c69519aba"
server_key_iv: "ae5d22f3761f9d55c0b13d6ca6bf8060"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.097184 8997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "26c6eb588e9146c19280f01845c768ff"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "b2728b542ed8144fe6e1e98c69519aba"
server_key_iv: "ae5d22f3761f9d55c0b13d6ca6bf8060"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.101019 8997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:57.103576 9011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:57.104945 8997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:11:57.105072 8997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "26c6eb588e9146c19280f01845c768ff"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "b2728b542ed8144fe6e1e98c69519aba"
server_key_iv: "ae5d22f3761f9d55c0b13d6ca6bf8060"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.105193 8997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:57.109843 8997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:57.110445 8997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:57.110627 8997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:57.111224 8997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:11:57.112241 8997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:11:57.112314 8997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:57.112386 8997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:11:57.112428 8997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:57.121366 8997 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:33985
I20260620 07:11:57.121387 9124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:33985 every 8 connection(s)
I20260620 07:11:57.122291 8997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:11:57.125193 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 8997
I20260620 07:11:57.125382 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:11:57.125677 4052 external_mini_cluster.cc:1428] Setting key 9858a17e04f23e65cccbc3a6437bb090
I20260620 07:11:57.129032 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.124186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:52855 (local address 127.3.245.62:35921)
0620 07:11:57.124385 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.124391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.125678 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 07:11:57.127312 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.127315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.127360 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.127449 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.128608 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.128652 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.128655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.128668 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.128690 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.128701 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.128775 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.128898 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.128934 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:11:57.129977 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.124803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35921 (local address 127.3.245.3:52855)
0620 07:11:57.125517 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.125566 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.127100 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.127587 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.127593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.127681 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.128357 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.128358 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.128400 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.128471 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.128771 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.128777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.128781 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.128924 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.129168 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":124,"threads_started":1}
I20260620 07:11:57.131151 9125 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35921
I20260620 07:11:57.131356 9125 heartbeater.cc:464] Registering TS with master...
I20260620 07:11:57.131708 9125 heartbeater.cc:511] Master 127.3.245.62:35921 requested a full tablet report, sending...
I20260620 07:11:57.132719 8675 ts_manager.cc:194] Registered new tserver with Master: 26c6eb588e9146c19280f01845c768ff (127.3.245.3:33985)
I20260620 07:11:57.140414 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:11:57.144196 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.142549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42894 (local address 127.3.245.62:35921)
0620 07:11:57.142682 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.142686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.142813 (+ 127us) server_negotiation.cc:407] Connection header received
0620 07:11:57.142924 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.142926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.142960 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.143026 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.143738 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.143740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.143742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.143752 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.143769 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.143776 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.143911 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.143997 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.144030 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:11:57.147661 8675 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:11:57.150943 8675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:11:57.163424 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.161017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.3.245.1:33301)
0620 07:11:57.161826 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.161840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.161866 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:11:57.161921 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.161927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.162101 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.162235 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.162863 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.162868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.162877 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.162885 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.162917 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.162935 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.163062 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.163193 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.163271 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":125,"threads_started":1}
I20260620 07:11:57.163408 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.160830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:33301 (local address 127.0.0.1:47934)
0620 07:11:57.161099 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.161115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.161219 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.162222 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.162226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.162242 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.162663 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.162664 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.162680 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.162732 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.163035 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.163044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.163050 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.163182 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.163258 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":136,"threads_started":1}
I20260620 07:11:57.164203 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.160231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.3.245.3:33985)
0620 07:11:57.160704 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.160716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.160745 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:11:57.160824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.160829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.161032 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.161180 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.163083 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.163091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.163107 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.163123 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.163331 (+ 208us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.163366 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.163451 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.163945 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.164027 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260620 07:11:57.164352 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.161208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:34773 (local address 127.0.0.1:33766)
0620 07:11:57.161558 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.161570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.161635 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.162454 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.162458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.162472 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.163579 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.163582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.163597 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.163634 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.164122 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.164125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.164129 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.164199 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.164233 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":83,"threads_started":1}
I20260620 07:11:57.164680 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.160053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:33985 (local address 127.0.0.1:43224)
0620 07:11:57.160401 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.160441 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.160568 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.161762 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.161770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.161849 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.162907 (+ 1058us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.162908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.162920 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.162945 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.163775 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.163778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.163781 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.164453 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.164501 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260620 07:11:57.165149 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.161521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.3.245.2:34773)
0620 07:11:57.161993 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.162003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.162023 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:11:57.162067 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.162071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.162207 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.162318 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.163871 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.163878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.163890 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.163903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.163946 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.163977 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.164079 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.164877 (+ 798us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.164958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":102,"threads_started":1}
I20260620 07:11:57.166978 8797 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b (DEFAULT_TABLE table=test-table [id=4a428aaf84ad4f46997b21faed53c8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:57.167311 9059 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b (DEFAULT_TABLE table=test-table [id=4a428aaf84ad4f46997b21faed53c8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:57.168205 8797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce30f59f3d3404da570d9c082d23f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:57.167898 8924 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b (DEFAULT_TABLE table=test-table [id=4a428aaf84ad4f46997b21faed53c8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:11:57.168360 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce30f59f3d3404da570d9c082d23f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:57.168844 8924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce30f59f3d3404da570d9c082d23f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:57.176604 9143 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Bootstrap starting.
I20260620 07:11:57.177201 9144 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Bootstrap starting.
I20260620 07:11:57.179198 9143 tablet_bootstrap.cc:654] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:57.179418 9144 tablet_bootstrap.cc:654] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:57.179880 9142 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Bootstrap starting.
I20260620 07:11:57.180167 9143 log.cc:824] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:57.180200 9144 log.cc:824] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:57.182161 9142 tablet_bootstrap.cc:654] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:57.182605 9144 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: No bootstrap required, opened a new log
I20260620 07:11:57.182813 9144 ts_tablet_manager.cc:1397] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:11:57.183036 9142 log.cc:824] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:57.183706 9143 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: No bootstrap required, opened a new log
I20260620 07:11:57.183895 9143 ts_tablet_manager.cc:1397] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 07:11:57.185101 9142 tablet_bootstrap.cc:492] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: No bootstrap required, opened a new log
I20260620 07:11:57.185284 9142 ts_tablet_manager.cc:1397] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:11:57.187489 9143 raft_consensus.cc:348] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.187547 9144 raft_consensus.cc:348] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.187760 9143 raft_consensus.cc:374] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:57.187820 9143 raft_consensus.cc:729] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9aec33acac24290b27ce077c02fff3c, State: Initialized, Role: FOLLOWER
I20260620 07:11:57.187814 9144 raft_consensus.cc:374] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:57.187875 9144 raft_consensus.cc:729] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6eb588e9146c19280f01845c768ff, State: Initialized, Role: FOLLOWER
I20260620 07:11:57.188305 9143 consensus_queue.cc:260] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.188434 9144 consensus_queue.cc:260] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.189124 8863 heartbeater.cc:503] Master 127.3.245.62:35921 was elected leader, sending a full tablet report...
I20260620 07:11:57.189545 9143 ts_tablet_manager.cc:1428] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 07:11:57.189875 9142 raft_consensus.cc:348] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.190147 9142 raft_consensus.cc:374] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:57.190151 9144 ts_tablet_manager.cc:1428] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 07:11:57.190203 9142 raft_consensus.cc:729] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407e429a98004d41b5406c23740c7e6a, State: Initialized, Role: FOLLOWER
I20260620 07:11:57.190490 9125 heartbeater.cc:503] Master 127.3.245.62:35921 was elected leader, sending a full tablet report...
I20260620 07:11:57.190694 9142 consensus_queue.cc:260] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.191495 8994 heartbeater.cc:503] Master 127.3.245.62:35921 was elected leader, sending a full tablet report...
I20260620 07:11:57.191645 9142 ts_tablet_manager.cc:1428] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 07:11:57.191968 8995 tablet.cc:2369] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:11:57.250778 8864 tablet.cc:2369] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:11:57.301453 9148 raft_consensus.cc:482] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:11:57.301733 9148 raft_consensus.cc:504] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.302927 9148 leader_election.cc:288] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26c6eb588e9146c19280f01845c768ff (127.3.245.3:33985), 407e429a98004d41b5406c23740c7e6a (127.3.245.2:34773)
I20260620 07:11:57.305308 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.303284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:52221 (local address 127.3.245.3:33985)
0620 07:11:57.303483 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.303488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.303640 (+ 152us) server_negotiation.cc:407] Connection header received
0620 07:11:57.303802 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.303808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.303849 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.303936 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.304875 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.304878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.304881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.304894 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.304915 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.304928 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.305017 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.305100 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.305160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 07:11:57.305335 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.303174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:33985 (local address 127.3.245.1:52221)
0620 07:11:57.303540 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.303554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.303688 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.303974 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.303977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.303992 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.304704 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.304706 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.304716 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.304741 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.305011 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.305014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.305017 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.305112 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.305153 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":108,"threads_started":1}
I20260620 07:11:57.305467 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.303741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:33313 (local address 127.3.245.2:34773)
0620 07:11:57.303878 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.303882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.304052 (+ 170us) server_negotiation.cc:407] Connection header received
0620 07:11:57.304129 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.304133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.304179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.304271 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.305007 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.305011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.305014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.305027 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.305048 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.305062 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.305148 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.305246 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.305350 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:11:57.305835 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.303615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:34773 (local address 127.3.245.1:33313)
0620 07:11:57.303961 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.303981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.304056 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.304301 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.304305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.304321 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:11:57.304792 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:11:57.304794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:11:57.304804 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:11:57.304867 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:11:57.305151 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:11:57.305153 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:11:57.305156 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 07:11:57.305235 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:11:57.305267 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":66,"threads_started":1}
I20260620 07:11:57.306201 9079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce30f59f3d3404da570d9c082d23f3b" candidate_uuid: "e9aec33acac24290b27ce077c02fff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c6eb588e9146c19280f01845c768ff" is_pre_election: true
I20260620 07:11:57.306401 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce30f59f3d3404da570d9c082d23f3b" candidate_uuid: "e9aec33acac24290b27ce077c02fff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407e429a98004d41b5406c23740c7e6a" is_pre_election: true
I20260620 07:11:57.306583 9079 raft_consensus.cc:2454] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9aec33acac24290b27ce077c02fff3c in term 0.
I20260620 07:11:57.306625 8946 raft_consensus.cc:2454] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9aec33acac24290b27ce077c02fff3c in term 0.
I20260620 07:11:57.307098 8753 leader_election.cc:302] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 407e429a98004d41b5406c23740c7e6a, e9aec33acac24290b27ce077c02fff3c; no voters:
I20260620 07:11:57.307345 9148 raft_consensus.cc:2781] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:11:57.307431 9148 raft_consensus.cc:482] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:11:57.307483 9148 raft_consensus.cc:3037] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:57.308589 9148 raft_consensus.cc:504] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.309026 9148 leader_election.cc:288] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [CANDIDATE]: Term 1 election: Requested vote from peers 26c6eb588e9146c19280f01845c768ff (127.3.245.3:33985), 407e429a98004d41b5406c23740c7e6a (127.3.245.2:34773)
I20260620 07:11:57.309425 9079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce30f59f3d3404da570d9c082d23f3b" candidate_uuid: "e9aec33acac24290b27ce077c02fff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c6eb588e9146c19280f01845c768ff"
I20260620 07:11:57.309440 8946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce30f59f3d3404da570d9c082d23f3b" candidate_uuid: "e9aec33acac24290b27ce077c02fff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407e429a98004d41b5406c23740c7e6a"
I20260620 07:11:57.309530 9079 raft_consensus.cc:3037] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:57.309594 8946 raft_consensus.cc:3037] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:57.310590 9079 raft_consensus.cc:2454] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9aec33acac24290b27ce077c02fff3c in term 1.
I20260620 07:11:57.310729 8946 raft_consensus.cc:2454] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9aec33acac24290b27ce077c02fff3c in term 1.
I20260620 07:11:57.310979 8753 leader_election.cc:302] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26c6eb588e9146c19280f01845c768ff, e9aec33acac24290b27ce077c02fff3c; no voters:
I20260620 07:11:57.311141 9148 raft_consensus.cc:2781] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:57.311354 9148 raft_consensus.cc:686] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 1 LEADER]: Becoming Leader. State: Replica: e9aec33acac24290b27ce077c02fff3c, State: Running, Role: LEADER
I20260620 07:11:57.311734 9148 consensus_queue.cc:237] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } }
I20260620 07:11:57.314967 8672 catalog_manager.cc:5495] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c reported cstate change: term changed from 0 to 1, leader changed from <none> to e9aec33acac24290b27ce077c02fff3c (127.3.245.1). New cstate: current_term: 1 leader_uuid: "e9aec33acac24290b27ce077c02fff3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9aec33acac24290b27ce077c02fff3c" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 33301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 } health_report { overall_health: UNKNOWN } } }
I20260620 07:11:57.326964 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.325212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.3.245.1:33301)
0620 07:11:57.325346 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.325350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.325364 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:11:57.325534 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.325536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.325599 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.325674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.326528 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.326532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.326535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.326548 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.326566 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.326575 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.326683 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.326766 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.326811 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:11:57.333868 8946 raft_consensus.cc:1261] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 1 FOLLOWER]: Refusing update from remote peer e9aec33acac24290b27ce077c02fff3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:57.333868 9079 raft_consensus.cc:1261] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 1 FOLLOWER]: Refusing update from remote peer e9aec33acac24290b27ce077c02fff3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:11:57.334726 9148 consensus_queue.cc:1035] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c6eb588e9146c19280f01845c768ff" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:57.335029 9153 consensus_queue.cc:1035] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "407e429a98004d41b5406c23740c7e6a" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 34773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:11:57.346969 9157 mvcc.cc:205] Tried to move back new op lower bound from 7298824262990905344 to 7298824262911651840. Current Snapshot: MvccSnapshot[applied={T|T < 7298824262990905344}]
I20260620 07:11:57.347174 9158 mvcc.cc:205] Tried to move back new op lower bound from 7298824262990905344 to 7298824262911651840. Current Snapshot: MvccSnapshot[applied={T|T < 7298824262990905344}]
I20260620 07:11:57.352922 8675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:57.353101 8675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:11:57.355774 8675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4cc9571861e4d32bb445d00deda3f42: Sending DeleteTablet for 3 replicas of tablet 2ce30f59f3d3404da570d9c082d23f3b
I20260620 07:11:57.356745 9059 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:57 UTC) from {username='slave'} at 127.0.0.1:43224
I20260620 07:11:57.356740 8797 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:57 UTC) from {username='slave'} at 127.0.0.1:47934
I20260620 07:11:57.356917 8924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce30f59f3d3404da570d9c082d23f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:11:57 UTC) from {username='slave'} at 127.0.0.1:33766
I20260620 07:11:57.357299 9165 tablet_replica.cc:330] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c: stopping tablet replica
I20260620 07:11:57.357299 9166 tablet_replica.cc:330] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: stopping tablet replica
I20260620 07:11:57.357642 9166 raft_consensus.cc:2229] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:57.357642 9165 raft_consensus.cc:2229] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:11:57.357975 9166 raft_consensus.cc:2258] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:57.358008 9165 raft_consensus.cc:2258] T 2ce30f59f3d3404da570d9c082d23f3b P e9aec33acac24290b27ce077c02fff3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:57.358345 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8735
I20260620 07:11:57.358919 9167 tablet_replica.cc:330] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: stopping tablet replica
I20260620 07:11:57.359242 9167 raft_consensus.cc:2229] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:11:57.359539 9167 raft_consensus.cc:2258] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:11:57.361460 9167 ts_tablet_manager.cc:1905] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:57.361840 9166 ts_tablet_manager.cc:1905] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:11:57.363788 9167 ts_tablet_manager.cc:1918] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:57.363912 9167 log.cc:1196] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/2ce30f59f3d3404da570d9c082d23f3b
I20260620 07:11:57.364190 9167 ts_tablet_manager.cc:1939] T 2ce30f59f3d3404da570d9c082d23f3b P 407e429a98004d41b5406c23740c7e6a: Deleting consensus metadata
I20260620 07:11:57.365228 9166 ts_tablet_manager.cc:1918] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:11:57.365336 9166 log.cc:1196] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/2ce30f59f3d3404da570d9c082d23f3b
I20260620 07:11:57.365670 9166 ts_tablet_manager.cc:1939] T 2ce30f59f3d3404da570d9c082d23f3b P 26c6eb588e9146c19280f01845c768ff: Deleting consensus metadata
I20260620 07:11:57.366040 8662 catalog_manager.cc:4841] TS 407e429a98004d41b5406c23740c7e6a (127.3.245.2:34773): tablet 2ce30f59f3d3404da570d9c082d23f3b (table test-table [id=4a428aaf84ad4f46997b21faed53c8d1]) successfully deleted
I20260620 07:11:57.366410 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8866
W20260620 07:11:57.366385 8662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:33301 (error 108)
I20260620 07:11:57.368112 8662 catalog_manager.cc:4841] TS 26c6eb588e9146c19280f01845c768ff (127.3.245.3:33985): tablet 2ce30f59f3d3404da570d9c082d23f3b (table test-table [id=4a428aaf84ad4f46997b21faed53c8d1]) successfully deleted
I20260620 07:11:57.368886 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.368463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:33301 (local address 127.0.0.1:47940)
0620 07:11:57.368665 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.368770 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:33301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:11:57.369086 8662 catalog_manager.cc:4584] TS e9aec33acac24290b27ce077c02fff3c (127.3.245.1:33301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ce30f59f3d3404da570d9c082d23f3b: Network error: Client connection negotiation failed: client connection to 127.3.245.1:33301: connect: Connection refused (error 111)
I20260620 07:11:57.372710 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8997
I20260620 07:11:57.378760 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 8644
2026-06-20T07:11:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1058 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 07:11:57.401281 4052 mini_oidc.cc:152] Starting JWKS server
I20260620 07:11:57.402952 4052 webserver.cc:460] Webserver started at https://127.0.0.1:38125/ using document root <none> and password file <none>
I20260620 07:11:57.403107 4052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:11:57.403407 4052 webserver.cc:460] Webserver started at http://127.0.0.1:46881/ using document root <none> and password file <none>
2026-06-20T07:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:57Z Disabled control of system clock
I20260620 07:11:57.413043 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:42385
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:42385
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/testchainca.cert with env {}
W20260620 07:11:57.530304 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:57.530597 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:57.530640 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:57.534727 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:57.534794 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:57.534811 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:57.534828 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:57.534843 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:11:57.534859 9174 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:11:57.534873 9174 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38125/jwks.json
I20260620 07:11:57.540604 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:42385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:57.541705 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:57.542695 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:57.551510 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:57.551488 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:57.551488 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:57.551540 9174 server_base.cc:1034] running on GCE node
I20260620 07:11:57.552337 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:57.553540 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:57.554730 9174 hybrid_clock.cc:648] HybridClock initialized: now 1781939517554699 us; error 43 us; skew 500 ppm
I20260620 07:11:57.556895 9174 webserver.cc:460] Webserver started at http://127.3.245.62:34147/ using document root <none> and password file <none>
I20260620 07:11:57.557515 9174 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:57.557593 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:57.557869 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:57.560233 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "d69042d353244f9284dfd0f987ff0e30"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "e16edd70e4b26f7843e27bdd3972dcd1"
server_key_iv: "fe3fb898110bf781a6941a780172bb72"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.560868 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "d69042d353244f9284dfd0f987ff0e30"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "e16edd70e4b26f7843e27bdd3972dcd1"
server_key_iv: "fe3fb898110bf781a6941a780172bb72"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.564785 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:11:57.567454 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:57.568969 9174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:57.569104 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "d69042d353244f9284dfd0f987ff0e30"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "e16edd70e4b26f7843e27bdd3972dcd1"
server_key_iv: "fe3fb898110bf781a6941a780172bb72"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.569223 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:57.599128 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:57.599879 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:57.600086 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:57.607707 9174 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:42385
I20260620 07:11:57.607722 9240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:42385 every 8 connection(s)
I20260620 07:11:57.608740 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:57.609757 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9174
I20260620 07:11:57.609867 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:57.610165 4052 external_mini_cluster.cc:1428] Setting key cb44f75ace98455269c851f71358f6fb
I20260620 07:11:57.612311 9241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:57.619009 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30: Bootstrap starting.
I20260620 07:11:57.619638 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.611511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35386 (local address 127.3.245.62:42385)
0620 07:11:57.612129 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.612144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.612178 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:11:57.612935 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.612953 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.613355 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.613630 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.613969 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.614960 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.616200 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.616816 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.616936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:57.617830 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.617835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.617848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.618237 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.618294 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.618314 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.618429 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.618557 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.618854 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":146,"threads_started":1}
I20260620 07:11:57.621564 9241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:57.622392 9241 log.cc:824] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:57.622649 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:11:57.624431 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30: No bootstrap required, opened a new log
I20260620 07:11:57.627816 9241 raft_consensus.cc:348] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } }
I20260620 07:11:57.628055 9241 raft_consensus.cc:374] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:57.628090 9241 raft_consensus.cc:729] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d69042d353244f9284dfd0f987ff0e30, State: Initialized, Role: FOLLOWER
I20260620 07:11:57.628454 9241 consensus_queue.cc:260] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } }
I20260620 07:11:57.628561 9241 raft_consensus.cc:388] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:57.628654 9241 raft_consensus.cc:482] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:57.628747 9241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:57.629616 9241 raft_consensus.cc:504] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } }
I20260620 07:11:57.629961 9241 leader_election.cc:302] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d69042d353244f9284dfd0f987ff0e30; no voters:
I20260620 07:11:57.630317 9241 leader_election.cc:288] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:57.630388 9246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:57.630594 9246 raft_consensus.cc:686] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [term 1 LEADER]: Becoming Leader. State: Replica: d69042d353244f9284dfd0f987ff0e30, State: Running, Role: LEADER
I20260620 07:11:57.630887 9246 consensus_queue.cc:237] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } }
I20260620 07:11:57.631640 9241 sys_catalog.cc:564] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:57.632582 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d69042d353244f9284dfd0f987ff0e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } } }
I20260620 07:11:57.632647 9248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d69042d353244f9284dfd0f987ff0e30. Latest consensus state: current_term: 1 leader_uuid: "d69042d353244f9284dfd0f987ff0e30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d69042d353244f9284dfd0f987ff0e30" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42385 } } }
I20260620 07:11:57.632735 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:57.632764 9248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:57.633162 9255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:57.636366 9255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:57.642812 9255 catalog_manager.cc:1347] Generated new cluster ID: 1622d3295f6e4c89a5640f1e7b5c4709
I20260620 07:11:57.642912 9255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:57.658825 9255 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:57.659821 9255 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:57.671209 9255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d69042d353244f9284dfd0f987ff0e30: Generated new TSK 0
I20260620 07:11:57.671978 9255 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:11:57.704777 9271 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 07:11:57.704895 9271 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:11:57.705523 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.687270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.3.245.62:42385)
0620 07:11:57.687419 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.687424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.687629 (+ 205us) server_negotiation.cc:407] Connection header received
0620 07:11:57.687839 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.687843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.687906 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.688029 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:11:57.688318 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.689171 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.690062 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.690408 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.690513 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:57.690835 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:11:57.705090 (+ 14255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:11:57.705380 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 07:11:57.705586 9244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 07:11:57.706666 9270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:11:57.687119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:42385 (local address 127.0.0.1:35390)
0620 07:11:57.687541 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:57.687555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:57.687670 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:57.688076 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:57.688079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:57.688094 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:11:57.688193 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:57.688198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.689281 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.689283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:57.689951 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:57.689959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.690522 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.690524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:57.690586 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:57.690634 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:11:57.705316 (+ 14682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 07:11:57.705549 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.245.62:42385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.245.62:42385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 07:11:57.707420 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9174
2026-06-20T07:11:57Z chronyd exiting
I20260620 07:11:57.745220 4052 test_util.cc:175] -----------------------------------------------
I20260620 07:11:57.745332 4052 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781939488540940-4052-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (345 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 07:11:57.746415 4052 mini_oidc.cc:152] Starting JWKS server
I20260620 07:11:57.747318 4052 webserver.cc:460] Webserver started at https://127.0.0.1:33727/ using document root <none> and password file <none>
I20260620 07:11:57.747452 4052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:11:57.747718 4052 webserver.cc:460] Webserver started at http://127.0.0.1:37703/ using document root <none> and password file <none>
2026-06-20T07:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:57Z Disabled control of system clock
I20260620 07:11:57.756738 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45297
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:37797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33727/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/testchainca.cert with env {}
W20260620 07:11:57.875108 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:57.875453 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:57.875499 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:57.879666 9277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:57.879750 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:57.879769 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:57.879789 9277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:57.879808 9277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:11:57.879828 9277 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:11:57.879873 9277 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33727/jwks.json
I20260620 07:11:57.885728 9277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:37797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33727/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:57.886912 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:57.887980 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:57.897747 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:57.897655 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:57.897799 9277 server_base.cc:1034] running on GCE node
W20260620 07:11:57.897655 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:57.898697 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:57.900012 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:57.901222 9277 hybrid_clock.cc:648] HybridClock initialized: now 1781939517901187 us; error 52 us; skew 500 ppm
I20260620 07:11:57.908399 9277 webserver.cc:460] Webserver started at http://127.3.245.62:44133/ using document root <none> and password file <none>
I20260620 07:11:57.909121 9277 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:57.909196 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:57.909430 9277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:57.911702 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "a5257f05c84d403286993470fb10e142"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "cc8a6e259b1727d21924fd680b40ad8c"
server_key_iv: "10ae3d60b1719204ec306d025ae2b62a"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.912323 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "a5257f05c84d403286993470fb10e142"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "cc8a6e259b1727d21924fd680b40ad8c"
server_key_iv: "10ae3d60b1719204ec306d025ae2b62a"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.916512 9277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:11:57.919252 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:57.920769 9277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:11:57.920946 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "a5257f05c84d403286993470fb10e142"
format_stamp: "Formatted at 2026-06-20 07:11:57 on dist-test-slave-ldfr"
server_key: "cc8a6e259b1727d21924fd680b40ad8c"
server_key_iv: "10ae3d60b1719204ec306d025ae2b62a"
server_key_version: "encryptionkey@0"
I20260620 07:11:57.921075 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:57.934755 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:57.935562 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:57.935801 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:57.944316 9277 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45297
I20260620 07:11:57.944461 9343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45297 every 8 connection(s)
I20260620 07:11:57.945485 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:57.949054 9344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:57.953065 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9277
I20260620 07:11:57.953212 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:57.953526 4052 external_mini_cluster.cc:1428] Setting key e6a0440fb13d0df8330ed742216a87a6
I20260620 07:11:57.955612 9344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: Bootstrap starting.
I20260620 07:11:57.958036 9344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:57.958848 9344 log.cc:824] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:57.961335 9344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: No bootstrap required, opened a new log
I20260620 07:11:57.963717 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:57.955008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.3.245.62:45297)
0620 07:11:57.955765 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:57.955820 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:57.955861 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:11:57.956572 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:57.956590 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:57.957095 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:57.957445 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:57.957902 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.958941 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.960059 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:57.960536 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:57.960745 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:57.961714 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:57.961719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:57.961734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:57.962160 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:57.962220 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:57.962240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:57.962329 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:57.962574 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 07:11:57.962903 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":161,"threads_started":1}
I20260620 07:11:57.965354 9344 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } }
I20260620 07:11:57.965633 9344 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:57.965710 9344 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5257f05c84d403286993470fb10e142, State: Initialized, Role: FOLLOWER
I20260620 07:11:57.966207 9344 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } }
I20260620 07:11:57.966398 9344 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:57.966470 9344 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:57.966580 9344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:57.966895 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:11:57.967737 9344 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } }
I20260620 07:11:57.968168 9344 leader_election.cc:302] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5257f05c84d403286993470fb10e142; no voters:
I20260620 07:11:57.968533 9344 leader_election.cc:288] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:57.968699 9349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:57.968897 9349 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [term 1 LEADER]: Becoming Leader. State: Replica: a5257f05c84d403286993470fb10e142, State: Running, Role: LEADER
I20260620 07:11:57.969208 9349 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } }
I20260620 07:11:57.969923 9344 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:57.970429 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5257f05c84d403286993470fb10e142. Latest consensus state: current_term: 1 leader_uuid: "a5257f05c84d403286993470fb10e142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } } }
I20260620 07:11:57.970549 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:57.970954 9351 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5257f05c84d403286993470fb10e142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5257f05c84d403286993470fb10e142" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45297 } } }
I20260620 07:11:57.971097 9351 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142 [sys.catalog]: This master's current role is: LEADER
W20260620 07:11:57.975502 9364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:11:57.975618 9364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:11:57.975778 9353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:57.976574 9353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:57.982761 9353 catalog_manager.cc:1347] Generated new cluster ID: 185168a131b54084a91d1e8d071e929a
I20260620 07:11:57.982867 9353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:57.990209 9353 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:57.991253 9353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:57.996579 9353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5257f05c84d403286993470fb10e142: Generated new TSK 0
I20260620 07:11:57.997459 9353 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 07:11:58.027455 9374 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 07:11:58.027580 9374 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 07:11:58.028220 9347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:58.011544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.3.245.62:45297)
0620 07:11:58.011689 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:58.011694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:58.011846 (+ 152us) server_negotiation.cc:407] Connection header received
0620 07:11:58.011988 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:58.011999 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:58.012059 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:58.012184 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:11:58.012398 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.013289 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.014181 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.014544 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.014649 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.015024 (+ 375us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:11:58.027773 (+ 12749us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33727/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:11:58.028062 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33727/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:11:58.028291 9347 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33727/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 07:11:58.028458 9373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:11:58.011444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45297 (local address 127.0.0.1:50012)
0620 07:11:58.011753 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:58.011772 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:58.011868 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:58.012187 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:58.012189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:58.012195 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:11:58.012293 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.012297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.013438 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.013441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.014067 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.014075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.014693 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.014696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.014797 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.014822 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:11:58.027999 (+ 13177us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33727/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 07:11:58.028183 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.245.62:45297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33727/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260620 07:11:58.029129 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9277
2026-06-20T07:11:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 07:11:58.069398 4052 mini_oidc.cc:152] Starting JWKS server
I20260620 07:11:58.070266 4052 webserver.cc:460] Webserver started at https://127.0.0.1:43331/ using document root <none> and password file <none>
I20260620 07:11:58.070392 4052 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 07:11:58.070648 4052 webserver.cc:460] Webserver started at http://127.0.0.1:34239/ using document root <none> and password file <none>
2026-06-20T07:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:58Z Disabled control of system clock
I20260620 07:11:58.081297 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:38901
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:38901
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43331/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:11:58.193490 9380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:58.193759 9380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:58.193799 9380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:58.197924 9380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:58.197980 9380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:58.197995 9380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:58.198012 9380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:58.198027 9380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:11:58.198089 9380 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:11:58.198108 9380 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43331/jwks.json
I20260620 07:11:58.203006 9380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:38901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43331/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:58.204106 9380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:58.205523 9380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:58.214213 9385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:58.214242 9388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:58.214422 9386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:58.214286 9380 server_base.cc:1034] running on GCE node
I20260620 07:11:58.215062 9380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:58.216459 9380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:58.217664 9380 hybrid_clock.cc:648] HybridClock initialized: now 1781939518217626 us; error 53 us; skew 500 ppm
I20260620 07:11:58.219653 9380 webserver.cc:460] Webserver started at http://127.3.245.62:33093/ using document root <none> and password file <none>
I20260620 07:11:58.220250 9380 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:58.220343 9380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:58.220577 9380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:58.222651 9380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "40e57f7f9468438eab2ea12c85fc190f"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "3c0d76655e9e6afd00106a30307d7904"
server_key_iv: "300301b4ad20f1e78589d0f701514c67"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.223222 9380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "40e57f7f9468438eab2ea12c85fc190f"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "3c0d76655e9e6afd00106a30307d7904"
server_key_iv: "300301b4ad20f1e78589d0f701514c67"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.227084 9380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:11:58.229568 9394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:58.230803 9380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:11:58.230947 9380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "40e57f7f9468438eab2ea12c85fc190f"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "3c0d76655e9e6afd00106a30307d7904"
server_key_iv: "300301b4ad20f1e78589d0f701514c67"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.231062 9380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:58.244714 9380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:58.245419 9380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:58.245615 9380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:58.252732 9380 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:38901
I20260620 07:11:58.252732 9446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:38901 every 8 connection(s)
I20260620 07:11:58.253805 9380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:58.257081 9447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:58.257288 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9380
I20260620 07:11:58.257411 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:58.257701 4052 external_mini_cluster.cc:1428] Setting key 16275c4f74b440d72a3a401a1a57532e
I20260620 07:11:58.263870 9447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f: Bootstrap starting.
I20260620 07:11:58.266426 9447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:58.266810 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:58.259007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36418 (local address 127.3.245.62:38901)
0620 07:11:58.259610 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:58.259626 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:58.259660 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:11:58.260332 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:58.260348 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:58.260811 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:58.261097 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:58.261421 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.262343 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.263503 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.264003 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.264139 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.265050 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:58.265055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:58.265067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:58.265447 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:58.265500 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:58.265520 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:58.265595 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:58.265758 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 07:11:58.266070 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":131,"threads_started":1}
I20260620 07:11:58.267244 9447 log.cc:824] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:58.269323 9447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f: No bootstrap required, opened a new log
I20260620 07:11:58.269789 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:11:58.273420 9447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } }
I20260620 07:11:58.273639 9447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:58.273674 9447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e57f7f9468438eab2ea12c85fc190f, State: Initialized, Role: FOLLOWER
I20260620 07:11:58.274103 9447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } }
I20260620 07:11:58.274217 9447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:58.274302 9447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:58.274437 9447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:58.275350 9447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } }
I20260620 07:11:58.275679 9447 leader_election.cc:302] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e57f7f9468438eab2ea12c85fc190f; no voters:
I20260620 07:11:58.276041 9447 leader_election.cc:288] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:58.276248 9452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:58.276511 9452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [term 1 LEADER]: Becoming Leader. State: Replica: 40e57f7f9468438eab2ea12c85fc190f, State: Running, Role: LEADER
I20260620 07:11:58.276856 9452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } }
I20260620 07:11:58.277395 9447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:58.278787 9453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e57f7f9468438eab2ea12c85fc190f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } } }
I20260620 07:11:58.278918 9453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:58.278755 9454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e57f7f9468438eab2ea12c85fc190f. Latest consensus state: current_term: 1 leader_uuid: "40e57f7f9468438eab2ea12c85fc190f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e57f7f9468438eab2ea12c85fc190f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38901 } } }
I20260620 07:11:58.279072 9454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:58.279474 9463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:58.282420 9463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:58.287602 9463 catalog_manager.cc:1347] Generated new cluster ID: 73c9285733ff4b5c857eb3b129397b25
I20260620 07:11:58.287674 9463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:58.305541 9463 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:58.306828 9463 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:58.319936 9463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e57f7f9468438eab2ea12c85fc190f: Generated new TSK 0
I20260620 07:11:58.320852 9463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:11:58.338546 9450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:58.333295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.3.245.62:38901)
0620 07:11:58.333476 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:58.333482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:58.333643 (+ 161us) server_negotiation.cc:407] Connection header received
0620 07:11:58.333848 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:58.333851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:58.333899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:58.334017 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:11:58.334261 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.334945 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.335933 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.336168 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.336278 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.336583 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:11:58.338046 (+ 1463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43331/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:11:58.338277 (+ 231us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36434: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43331/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:11:58.338646 9450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36434: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43331/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 07:11:58.340801 9476 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:11:58.333197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:38901 (local address 127.0.0.1:36434)
0620 07:11:58.333552 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:58.333564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:58.333694 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:58.334019 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:58.334022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:58.334030 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:11:58.334144 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.334152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.335058 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.335064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.335811 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.335820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.336288 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.336292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.336376 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.336408 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:11:58.338243 (+ 1835us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43331/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 07:11:58.340591 (+ 2348us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.245.62:38901: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43331/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":2126,"thread_start_us":110,"threads_started":1}
I20260620 07:11:58.341580 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9380
E20260620 07:11:58.350835 9477 webserver.cc:533] Webserver: sslize failed:
E20260620 07:11:58.350914 9477 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T07:11:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T07:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:11:58Z Disabled control of system clock
I20260620 07:11:58.389631 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46585
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:46585
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/testchainca.cert with env {}
W20260620 07:11:58.501804 9481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:11:58.502085 9481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:11:58.502125 9481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:11:58.506246 9481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:11:58.506310 9481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:11:58.506326 9481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:11:58.506343 9481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:11:58.506358 9481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 07:11:58.506377 9481 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 07:11:58.506392 9481 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 07:11:58.511906 9481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:46585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:11:58.513032 9481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:11:58.514029 9481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:11:58.522894 9486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:58.522910 9487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:11:58.522954 9489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:11:58.522972 9481 server_base.cc:1034] running on GCE node
I20260620 07:11:58.523839 9481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:11:58.525102 9481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:11:58.526342 9481 hybrid_clock.cc:648] HybridClock initialized: now 1781939518526314 us; error 38 us; skew 500 ppm
I20260620 07:11:58.528208 9481 webserver.cc:460] Webserver started at http://127.3.245.62:35103/ using document root <none> and password file <none>
I20260620 07:11:58.528923 9481 fs_manager.cc:359] Metadata directory not provided
I20260620 07:11:58.529017 9481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:11:58.529258 9481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:11:58.531325 9481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "379b45c816ad4805ad8e9ef760b8def6"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "e7599f36af43b41edb273c4a6a7742e1"
server_key_iv: "3af17682f6099b90a735fe42fe7fff25"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.531899 9481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "379b45c816ad4805ad8e9ef760b8def6"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "e7599f36af43b41edb273c4a6a7742e1"
server_key_iv: "3af17682f6099b90a735fe42fe7fff25"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.535645 9481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:11:58.538166 9495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:11:58.539449 9481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 07:11:58.539618 9481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "379b45c816ad4805ad8e9ef760b8def6"
format_stamp: "Formatted at 2026-06-20 07:11:58 on dist-test-slave-ldfr"
server_key: "e7599f36af43b41edb273c4a6a7742e1"
server_key_iv: "3af17682f6099b90a735fe42fe7fff25"
server_key_version: "encryptionkey@0"
I20260620 07:11:58.539767 9481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:11:58.553335 9481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:11:58.554066 9481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:11:58.554267 9481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:11:58.561009 9481 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:46585
I20260620 07:11:58.561033 9547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:46585 every 8 connection(s)
I20260620 07:11:58.561931 9481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:11:58.565251 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9481
I20260620 07:11:58.565356 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:11:58.565434 9548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:11:58.565671 4052 external_mini_cluster.cc:1428] Setting key cd73b51c85699e34f10d1660405d68cb
I20260620 07:11:58.573472 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6: Bootstrap starting.
I20260620 07:11:58.576349 9548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6: Neither blocks nor log segments found. Creating new log.
I20260620 07:11:58.577301 9548 log.cc:824] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:11:58.577355 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:58.567095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.3.245.62:46585)
0620 07:11:58.567850 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:58.567870 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:58.567915 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:11:58.569316 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:58.569343 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:58.569905 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:58.570294 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:11:58.570773 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.572444 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.573440 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.574168 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.574337 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.575170 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:11:58.575177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:11:58.575193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:11:58.575719 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:11:58.575797 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:11:58.575825 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:11:58.575925 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:11:58.576033 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:11:58.576333 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260620 07:11:58.579511 9548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6: No bootstrap required, opened a new log
I20260620 07:11:58.580205 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:11:58.583632 9548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } }
I20260620 07:11:58.583874 9548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:11:58.583940 9548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379b45c816ad4805ad8e9ef760b8def6, State: Initialized, Role: FOLLOWER
I20260620 07:11:58.584434 9548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } }
I20260620 07:11:58.584568 9548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:11:58.584668 9548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:11:58.584787 9548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:11:58.585966 9548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } }
I20260620 07:11:58.586342 9548 leader_election.cc:302] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 379b45c816ad4805ad8e9ef760b8def6; no voters:
I20260620 07:11:58.586632 9548 leader_election.cc:288] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:11:58.586788 9553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:11:58.587054 9553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [term 1 LEADER]: Becoming Leader. State: Replica: 379b45c816ad4805ad8e9ef760b8def6, State: Running, Role: LEADER
I20260620 07:11:58.587378 9553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } }
I20260620 07:11:58.587996 9548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:11:58.589267 9555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 379b45c816ad4805ad8e9ef760b8def6. Latest consensus state: current_term: 1 leader_uuid: "379b45c816ad4805ad8e9ef760b8def6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } } }
I20260620 07:11:58.589396 9555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:58.589836 9562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:11:58.590126 9554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "379b45c816ad4805ad8e9ef760b8def6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379b45c816ad4805ad8e9ef760b8def6" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46585 } } }
I20260620 07:11:58.590237 9554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6 [sys.catalog]: This master's current role is: LEADER
I20260620 07:11:58.593472 9562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:11:58.599306 9562 catalog_manager.cc:1347] Generated new cluster ID: bedb45d11d584ac7916a45895bfe8f28
I20260620 07:11:58.599373 9562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:11:58.623034 9562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:11:58.624414 9562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:11:58.632650 9562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 379b45c816ad4805ad8e9ef760b8def6: Generated new TSK 0
I20260620 07:11:58.633594 9562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:11:58.701926 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:11:58.651234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.3.245.62:46585)
0620 07:11:58.651392 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:11:58.651397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:11:58.651640 (+ 243us) server_negotiation.cc:407] Connection header received
0620 07:11:58.651792 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:11:58.651795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:11:58.651855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:11:58.651979 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0620 07:11:58.652279 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.653155 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.654080 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.654450 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.654562 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.654857 (+ 295us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 07:11:58.701412 (+ 46555us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:11:58.701745 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 07:11:58.702021 9551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36804: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 07:11:58.702112 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:11:58.651151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46585 (local address 127.0.0.1:36804)
0620 07:11:58.651545 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 07:11:58.651557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:11:58.651656 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:11:58.651984 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:11:58.651986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:11:58.651993 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 07:11:58.652088 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.652093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.653299 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.653303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.653960 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:11:58.653968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:11:58.654571 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:11:58.654575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:11:58.654649 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:11:58.654672 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 07:11:58.701675 (+ 47003us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 07:11:58.701910 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.245.62:46585: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":174,"threads_started":1}
I20260620 07:11:58.702775 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9481
2026-06-20T07:11:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (340 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:11:59 dist-test-slave-ldfr krb5kdc[9589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:11:59 dist-test-slave-ldfr krb5kdc[9589](info): set up 2 sockets
Jun 20 07:11:59 dist-test-slave-ldfr krb5kdc[9589](info): commencing operation
krb5kdc: starting...
W20260620 07:11:59.993009 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:12:00.124009 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35837
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:35837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:00.234098 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:00.234365 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:00.234406 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:00.238490 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:00.238554 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:00.238574 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:00.238590 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:00.238612 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:00.244076 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:35837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:00.245137 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:00.246152 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:00.254940 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:00.254966 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:00.254982 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:00.255093 9602 server_base.cc:1034] running on GCE node
I20260620 07:12:00.255707 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:00.256973 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:00.258174 9602 hybrid_clock.cc:648] HybridClock initialized: now 1781939520258146 us; error 43 us; skew 500 ppm
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:00.261018 9602 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:12:00.261891 9602 webserver.cc:460] Webserver started at http://127.3.245.62:38115/ using document root <none> and password file <none>
I20260620 07:12:00.262409 9602 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:00.262466 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:00.262626 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:00.264746 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "37c86e124a064b9f928ea2d4d9df30ff"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "24f5391c265140c65329d78dd505affc"
server_key_iv: "11149031505a855420c32158257d453b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.265257 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "37c86e124a064b9f928ea2d4d9df30ff"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "24f5391c265140c65329d78dd505affc"
server_key_iv: "11149031505a855420c32158257d453b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.268981 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:12:00.271394 9617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.272704 9602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:12:00.272805 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "37c86e124a064b9f928ea2d4d9df30ff"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "24f5391c265140c65329d78dd505affc"
server_key_iv: "11149031505a855420c32158257d453b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.272915 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:00.292186 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:00.295231 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:00.295401 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:00.302619 9602 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:35837
I20260620 07:12:00.302644 9669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:35837 every 8 connection(s)
I20260620 07:12:00.303610 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:00.306475 9670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:00.310623 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9602
I20260620 07:12:00.310741 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:00.311044 4052 external_mini_cluster.cc:1428] Setting key 0edf13360c7b6aec7903fda7ff2f85d6
I20260620 07:12:00.312423 9670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff: Bootstrap starting.
I20260620 07:12:00.314914 9670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:00.315807 9670 log.cc:824] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:00.317868 9670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff: No bootstrap required, opened a new log
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:00.321926 9670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } }
I20260620 07:12:00.322147 9670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:00.322216 9670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c86e124a064b9f928ea2d4d9df30ff, State: Initialized, Role: FOLLOWER
I20260620 07:12:00.322722 9670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } }
I20260620 07:12:00.322876 9670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:00.322968 9670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:00.323072 9670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:00.324281 9670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } }
I20260620 07:12:00.324672 9670 leader_election.cc:302] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c86e124a064b9f928ea2d4d9df30ff; no voters:
I20260620 07:12:00.325040 9670 leader_election.cc:288] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:00.325157 9675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:00.325433 9675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [term 1 LEADER]: Becoming Leader. State: Replica: 37c86e124a064b9f928ea2d4d9df30ff, State: Running, Role: LEADER
I20260620 07:12:00.325748 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:00.312653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.3.245.62:35837)
0620 07:12:00.313111 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:00.313128 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:00.313162 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:12:00.313811 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:00.313829 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:00.314164 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:00.314456 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:00.315434 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.316474 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.317436 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.317933 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.318169 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:00.320659 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:00.320667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:00.320691 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:00.320717 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:00.322531 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.322991 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.322998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.323005 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.323160 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.323453 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.323457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.323458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.323977 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:00.324181 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:00.324394 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:12:00.324728 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":168,"threads_started":1}
I20260620 07:12:00.325754 9675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } }
I20260620 07:12:00.326323 9670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:00.327384 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c86e124a064b9f928ea2d4d9df30ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } } }
I20260620 07:12:00.327517 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:00.327754 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c86e124a064b9f928ea2d4d9df30ff. Latest consensus state: current_term: 1 leader_uuid: "37c86e124a064b9f928ea2d4d9df30ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c86e124a064b9f928ea2d4d9df30ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35837 } } }
I20260620 07:12:00.327843 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:00.328118 9684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:00.331354 9684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:00.337474 9684 catalog_manager.cc:1347] Generated new cluster ID: 6499b55736a748f5852384a1689c0cb3
I20260620 07:12:00.337558 9684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:00.368595 9684 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:00.369830 9684 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:00.382768 9684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c86e124a064b9f928ea2d4d9df30ff: Generated new TSK 0
I20260620 07:12:00.383546 9684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:12:00.424489 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:00.561580 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:00.561875 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:00.561959 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:00.566154 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:00.566232 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:00.566337 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:00.571985 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:00.573164 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:00.574239 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:00.584772 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:00.584824 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:00.584818 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:00.585170 9696 server_base.cc:1034] running on GCE node
I20260620 07:12:00.585614 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:00.586289 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:00.587471 9696 hybrid_clock.cc:648] HybridClock initialized: now 1781939520587448 us; error 33 us; skew 500 ppm
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:00.590715 9696 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:12:00.591795 9696 webserver.cc:460] Webserver started at http://127.3.245.1:41579/ using document root <none> and password file <none>
I20260620 07:12:00.592384 9696 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:00.592447 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:00.592734 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:00.594916 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "a911222f2efb492080e47eeaefbbad54"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "3f65be7b1f23c2ca7b989ced769b3388"
server_key_iv: "39b249d28160d42ec4c0aaa5632b2cf3"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.595510 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "a911222f2efb492080e47eeaefbbad54"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "3f65be7b1f23c2ca7b989ced769b3388"
server_key_iv: "39b249d28160d42ec4c0aaa5632b2cf3"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.599426 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:12:00.602100 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.603478 9696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:00.603619 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "a911222f2efb492080e47eeaefbbad54"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "3f65be7b1f23c2ca7b989ced769b3388"
server_key_iv: "39b249d28160d42ec4c0aaa5632b2cf3"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.603735 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:00.624235 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:00.627876 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:00.628088 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:00.628731 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:00.629587 9696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:00.629654 9696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.629727 9696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:00.629779 9696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.638618 9696 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:43849
I20260620 07:12:00.638638 9824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:43849 every 8 connection(s)
I20260620 07:12:00.639515 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:00.642565 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9696
I20260620 07:12:00.642697 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:00.642966 4052 external_mini_cluster.cc:1428] Setting key 154f94513509e8e051b2b6c75cb119a2
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:12:00.654402 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:00.641303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:54933 (local address 127.3.245.62:35837)
0620 07:12:00.641507 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:00.641511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:00.642318 (+ 807us) server_negotiation.cc:407] Connection header received
0620 07:12:00.643883 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:00.643886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:00.643927 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:00.643993 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:00.645291 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.645847 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.647023 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.647257 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.647367 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:00.650201 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:00.650205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:00.650208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:00.650242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:00.651964 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.652597 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.652601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.652603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.652776 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.653157 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.653160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.653161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.653367 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:00.653474 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:00.654036 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 07:12:00.654227 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:12:00.655382 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:00.641695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35837 (local address 127.3.245.1:54933)
0620 07:12:00.642156 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:00.642200 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:00.643673 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:00.644172 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:00.644179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:00.644528 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:00.645147 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:00.645161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.646006 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.646010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:00.646891 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:00.646900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.647408 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.647412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:00.647580 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:00.648197 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:00.648214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:00.649998 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:00.652113 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:00.652124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:00.652140 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:00.652486 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:00.652875 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:00.652878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:00.652880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:00.653033 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:00.653474 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:00.653482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:00.653758 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 07:12:00.653961 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 07:12:00.654320 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260620 07:12:00.656692 9825 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35837
I20260620 07:12:00.656986 9825 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:00.657474 9825 heartbeater.cc:511] Master 127.3.245.62:35837 requested a full tablet report, sending...
I20260620 07:12:00.659025 9634 ts_manager.cc:194] Registered new tserver with Master: a911222f2efb492080e47eeaefbbad54 (127.3.245.1:43849)
I20260620 07:12:00.660159 9634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:54933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:12:00.672824 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:00.790884 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:00.791173 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:00.791221 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:00.795679 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:00.795749 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:00.795810 9830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:00.801344 9830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:00.802366 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:00.803373 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:00.813336 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:00.813390 9838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:00.813398 9830 server_base.cc:1034] running on GCE node
W20260620 07:12:00.813344 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:00.814051 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:00.814757 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:00.815944 9830 hybrid_clock.cc:648] HybridClock initialized: now 1781939520815925 us; error 32 us; skew 500 ppm
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:00.818958 9830 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:12:00.819885 9830 webserver.cc:460] Webserver started at http://127.3.245.2:46103/ using document root <none> and password file <none>
I20260620 07:12:00.820457 9830 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:00.820547 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:00.820843 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:00.823282 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "b483f2ed0ddd475bb1e305b0169cecb4"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "ce368640083de80e4384e17ed6e037a3"
server_key_iv: "37ec286993f2d3106c0a1027148fcd9b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.823829 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "b483f2ed0ddd475bb1e305b0169cecb4"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "ce368640083de80e4384e17ed6e037a3"
server_key_iv: "37ec286993f2d3106c0a1027148fcd9b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.827908 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:12:00.830555 9845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.831854 9830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 07:12:00.831995 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "b483f2ed0ddd475bb1e305b0169cecb4"
format_stamp: "Formatted at 2026-06-20 07:12:00 on dist-test-slave-ldfr"
server_key: "ce368640083de80e4384e17ed6e037a3"
server_key_iv: "37ec286993f2d3106c0a1027148fcd9b"
server_key_version: "encryptionkey@0"
I20260620 07:12:00.832113 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:00.848497 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:00.852108 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:00.852345 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:00.853046 9830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:00.853945 9830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:00.854017 9830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.854089 9830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:00.854139 9830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:00.863003 9830 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:42177
I20260620 07:12:00.863021 9958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:42177 every 8 connection(s)
I20260620 07:12:00.863948 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:00.870075 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9830
I20260620 07:12:00.870246 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:00.870559 4052 external_mini_cluster.cc:1428] Setting key e41cac6a2217c22469aecb54fcca1d89
Jun 20 07:12:00 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:00.878648 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:00.865897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:59425 (local address 127.3.245.62:35837)
0620 07:12:00.866045 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:00.866050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:00.866894 (+ 844us) server_negotiation.cc:407] Connection header received
0620 07:12:00.868223 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:00.868226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:00.868274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:00.868353 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:00.869574 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.870106 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.871633 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.871912 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.872039 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:00.874944 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:00.874947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:00.874949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:00.874974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:00.876536 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.877098 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.877102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.877104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.877186 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:00.877533 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:00.877538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:00.877539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:00.877732 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:00.877841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:00.878353 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 07:12:00.878475 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:12:00.879456 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:00.866291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35837 (local address 127.3.245.2:59425)
0620 07:12:00.866726 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:00.866769 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:00.867995 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:00.868517 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:00.868523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:00.868938 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:00.869419 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:00.869432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.870246 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.870250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:00.871511 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:00.871520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:00.872075 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:00.872080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:00.872313 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:00.873009 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:00.873025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:00.874754 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:00.876685 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:00.876691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:00.876716 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:00.876989 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:00.877289 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:00.877292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:00.877294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:00.877439 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:00.877839 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:00.877846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:00.878108 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 07:12:00.878298 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:12:00.878622 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:12:00.880668 9959 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35837
I20260620 07:12:00.880939 9959 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:00.881451 9959 heartbeater.cc:511] Master 127.3.245.62:35837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:12:00.882658 9634 ts_manager.cc:194] Registered new tserver with Master: b483f2ed0ddd475bb1e305b0169cecb4 (127.3.245.2:42177)
I20260620 07:12:00.883306 9634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:59425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:12:00.901511 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:01.017943 9964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:01.018224 9964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:01.018273 9964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:01.022847 9964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:01.022908 9964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:01.023005 9964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:01.028548 9964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:01.029641 9964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:01.030658 9964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:01.040355 9972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:01.040436 9964 server_base.cc:1034] running on GCE node
W20260620 07:12:01.040360 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:01.040323 9970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:01.041025 9964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:01.041712 9964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:01.042892 9964 hybrid_clock.cc:648] HybridClock initialized: now 1781939521042859 us; error 37 us; skew 500 ppm
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.046314 9964 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:12:01.047273 9964 webserver.cc:460] Webserver started at http://127.3.245.3:40501/ using document root <none> and password file <none>
I20260620 07:12:01.047894 9964 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:01.047981 9964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:01.048221 9964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:01.050329 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "dbc3dcb39766472ba73e33377bffffd3"
format_stamp: "Formatted at 2026-06-20 07:12:01 on dist-test-slave-ldfr"
server_key: "fa7c97d4132b3e96f1bc1d41e768cbba"
server_key_iv: "0e2b98e96571904027352e6c92ff4b15"
server_key_version: "encryptionkey@0"
I20260620 07:12:01.050865 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "dbc3dcb39766472ba73e33377bffffd3"
format_stamp: "Formatted at 2026-06-20 07:12:01 on dist-test-slave-ldfr"
server_key: "fa7c97d4132b3e96f1bc1d41e768cbba"
server_key_iv: "0e2b98e96571904027352e6c92ff4b15"
server_key_version: "encryptionkey@0"
I20260620 07:12:01.054869 9964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 07:12:01.057528 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:01.058921 9964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:01.059032 9964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "dbc3dcb39766472ba73e33377bffffd3"
format_stamp: "Formatted at 2026-06-20 07:12:01 on dist-test-slave-ldfr"
server_key: "fa7c97d4132b3e96f1bc1d41e768cbba"
server_key_iv: "0e2b98e96571904027352e6c92ff4b15"
server_key_version: "encryptionkey@0"
I20260620 07:12:01.059137 9964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:01.093727 9964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:01.097121 9964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:01.097278 9964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:01.097796 9964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:01.098628 9964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:01.098676 9964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:01.098717 9964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:01.098732 9964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:01.107326 9964 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:44813
I20260620 07:12:01.107369 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:44813 every 8 connection(s)
I20260620 07:12:01.108321 9964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:01.108887 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 9964
I20260620 07:12:01.109015 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:01.109323 4052 external_mini_cluster.cc:1428] Setting key d056bdfe390114bcdb96376bcd42e190
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:01.123092 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:01.110203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:56335 (local address 127.3.245.62:35837)
0620 07:12:01.110334 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.110338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.111537 (+ 1199us) server_negotiation.cc:407] Connection header received
0620 07:12:01.113006 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.113009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.113056 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.113137 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.114540 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.115080 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.116211 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.116435 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.116517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:01.119239 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:01.119243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:01.119245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:01.119269 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:01.120911 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.121464 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.121467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.121468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.121540 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.121923 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.121925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.121926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.122094 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:01.122167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:01.122805 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 07:12:01.122934 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:12:01.124161 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:01.110773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35837 (local address 127.3.245.3:56335)
0620 07:12:01.111364 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:01.111423 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:01.112813 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:01.113515 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:01.113522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:01.113897 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:01.114395 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:01.114409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.115192 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.115195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:01.116099 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:01.116108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.116602 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.116606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:01.116861 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:01.117538 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:01.117556 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:01.119064 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:01.121024 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:01.121030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:01.121044 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:01.121357 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:01.121626 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:01.121628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:01.121630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:01.121798 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:01.122203 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:01.122210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:01.122491 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 07:12:01.122724 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 07:12:01.123107 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260620 07:12:01.125432 10093 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35837
I20260620 07:12:01.125675 10093 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:01.126199 10093 heartbeater.cc:511] Master 127.3.245.62:35837 requested a full tablet report, sending...
I20260620 07:12:01.127331 9634 ts_manager.cc:194] Registered new tserver with Master: dbc3dcb39766472ba73e33377bffffd3 (127.3.245.3:44813)
I20260620 07:12:01.127947 9634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:56335
I20260620 07:12:01.135900 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/test-admin.keytab.
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.154470 4052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 07:12:01.157500 10100 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:01.155881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.3.245.1:43849)
0620 07:12:01.156181 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.156189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.156221 (+ 32us) server_negotiation.cc:407] Connection header received
0620 07:12:01.156356 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.156362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.156552 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.156780 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.157293 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47456: BlockingRecv error: recv got EOF from 127.0.0.1:47456 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
W20260620 07:12:01.157578 4052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.158730 4052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.159705 4052 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:12:01.170171 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:01.160691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.3.245.1:43849)
0620 07:12:01.160808 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.160814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.160878 (+ 64us) server_negotiation.cc:407] Connection header received
0620 07:12:01.161043 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.161049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.161096 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.161252 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.161848 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.162632 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.163528 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.164034 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.164178 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:01.166606 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:01.166616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:01.166629 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:01.166652 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:01.168472 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.168961 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.168966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.168967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.169058 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.169298 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.169301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.169302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.169530 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:01.169702 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:01.169844 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:12:01.170047 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 07:12:01.176326 4052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc' of type FILE
W20260620 07:12:01.176350 10100 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:01.173717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.3.245.1:43849)
0620 07:12:01.173856 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.173862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.175430 (+ 1568us) server_negotiation.cc:407] Connection header received
0620 07:12:01.175544 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.175549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.175605 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.175687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.176205 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47474: BlockingRecv error: recv got EOF from 127.0.0.1:47474 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.177752 4052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.178881 4052 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:12:01.188493 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:01.179968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.3.245.1:43849)
0620 07:12:01.180086 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.180090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.180571 (+ 481us) server_negotiation.cc:407] Connection header received
0620 07:12:01.180647 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.180650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.180698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.180794 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.181405 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.181925 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.182737 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.182944 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.183016 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:01.185407 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:01.185409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:01.185412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:01.185434 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:01.186762 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.187255 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.187258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.187259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.187364 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.187632 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.187634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.187636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.187786 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:01.187904 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:01.188211 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 07:12:01.188371 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 07:12:01.191560 10100 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:01.190468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.3.245.1:43849)
0620 07:12:01.190582 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.190586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.190705 (+ 119us) server_negotiation.cc:407] Connection header received
0620 07:12:01.190794 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.190796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.190830 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.190899 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.191452 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47480: BlockingRecv error: recv got EOF from 127.0.0.1:47480 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 07:12:01.191823 4052 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781939488540940-4052-0/krb5kdc/krb5cc' of type FILE
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.192992 4052 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:01.193992 4052 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[9589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:12:01.203511 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:01.194913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.3.245.1:43849)
0620 07:12:01.195038 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:01.195043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:01.195234 (+ 191us) server_negotiation.cc:407] Connection header received
0620 07:12:01.195392 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:01.195396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:01.195448 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:01.195561 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:01.196121 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.196724 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.197581 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:01.197838 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:01.197953 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:01.200500 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:01.200503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:01.200505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:01.200528 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:01.201865 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.202286 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.202291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.202293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.202405 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:01.202678 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:01.202682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:01.202684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:01.202856 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:01.203009 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:01.203229 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 07:12:01.203349 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:12:01.204535 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9696
I20260620 07:12:01.210525 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9830
I20260620 07:12:01.215997 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9964
I20260620 07:12:01.221271 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 9602
2026-06-20T07:12:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2273 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[10113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[10113](info): set up 2 sockets
Jun 20 07:12:01 dist-test-slave-ldfr krb5kdc[10113](info): commencing operation
krb5kdc: starting...
W20260620 07:12:02.273228 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:02 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab.
Entry for principal oryx/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab.
I20260620 07:12:02.401157 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34251
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab
--principal=oryx/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:02.516705 10126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:02.517024 10126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:02.517067 10126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:02.521236 10126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:02.521325 10126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:02.521346 10126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:02.521364 10126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:02.521404 10126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:02.527158 10126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab
--principal=oryx/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:02.528285 10126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:02.529399 10126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:02.538400 10131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:02.538400 10134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:02.538395 10132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:02.538720 10126 server_base.cc:1034] running on GCE node
I20260620 07:12:02.539419 10126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:02.540669 10126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:02.541848 10126 hybrid_clock.cc:648] HybridClock initialized: now 1781939522541815 us; error 48 us; skew 500 ppm
Jun 20 07:12:02 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:02.544775 10126 init.cc:377] Logged in from keytab as oryx/127.3.245.62@KRBTEST.COM (short username oryx)
I20260620 07:12:02.545787 10126 webserver.cc:460] Webserver started at http://127.3.245.62:38581/ using document root <none> and password file <none>
I20260620 07:12:02.546330 10126 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:02.546391 10126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:02.546566 10126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:02.548791 10126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "7563d4cea48b47cdab79bde300cbf61d"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "c1a8271661ee5af8facae1654aeb40bf"
server_key_iv: "fd99c33eca8cbad900c84b1e63801051"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.549430 10126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "7563d4cea48b47cdab79bde300cbf61d"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "c1a8271661ee5af8facae1654aeb40bf"
server_key_iv: "fd99c33eca8cbad900c84b1e63801051"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.553673 10126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:02.556257 10141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:02.557691 10126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:02.557839 10126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "7563d4cea48b47cdab79bde300cbf61d"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "c1a8271661ee5af8facae1654aeb40bf"
server_key_iv: "fd99c33eca8cbad900c84b1e63801051"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.557966 10126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:02.570103 10126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:02.573657 10126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:02.573887 10126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:02.581542 10126 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:34251
I20260620 07:12:02.581933 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:34251 every 8 connection(s)
I20260620 07:12:02.582592 10126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:02.585546 10194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:02.587980 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10126
I20260620 07:12:02.588150 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:02.588450 4052 external_mini_cluster.cc:1428] Setting key eb820d3c4bc470d2d0e0cb4f60c16a95
I20260620 07:12:02.592453 10194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Bootstrap starting.
I20260620 07:12:02.595346 10194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:02.596295 10194 log.cc:824] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:02.598562 10194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: No bootstrap required, opened a new log
Jun 20 07:12:02 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:02.603120 10194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } }
I20260620 07:12:02.603487 10194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:02.603542 10194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7563d4cea48b47cdab79bde300cbf61d, State: Initialized, Role: FOLLOWER
I20260620 07:12:02.604146 10194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } }
I20260620 07:12:02.604341 10194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:02.604405 10194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:02.604491 10194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:02.605772 10194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } }
I20260620 07:12:02.605979 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:02.590067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.3.245.62:34251)
0620 07:12:02.590545 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:02.590563 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:02.590610 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:12:02.591496 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:02.591521 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:02.591981 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:02.592378 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:02.593276 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.594790 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.595727 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.596474 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.596672 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:02.599461 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:02.599468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:02.599486 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:02.599519 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:02.602146 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:02.602642 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:02.602649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:02.602656 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:02.602803 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:02.603279 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:02.603283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:02.603285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:02.603850 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:02.604073 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:02.604355 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 07:12:02.604911 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260620 07:12:02.606164 10194 leader_election.cc:302] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7563d4cea48b47cdab79bde300cbf61d; no voters:
I20260620 07:12:02.606562 10194 leader_election.cc:288] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:02.606731 10199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:02.607041 10199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 LEADER]: Becoming Leader. State: Replica: 7563d4cea48b47cdab79bde300cbf61d, State: Running, Role: LEADER
I20260620 07:12:02.607415 10199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } }
I20260620 07:12:02.607894 10194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:02.609319 10200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } }
I20260620 07:12:02.609314 10201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7563d4cea48b47cdab79bde300cbf61d. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } }
I20260620 07:12:02.609478 10200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:02.609478 10201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:02.609961 10209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:02.612857 10209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:02.618273 10209 catalog_manager.cc:1347] Generated new cluster ID: 5cb1b2eb4c974bd8a8f17109f2f80fc4
I20260620 07:12:02.618355 10209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:02.651829 10209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:02.652976 10209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:02.664832 10209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Generated new TSK 0
I20260620 07:12:02.665663 10209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
I20260620 07:12:02.699848 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:02.823624 10220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:02.823968 10220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:02.824018 10220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:02.828320 10220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:02.828387 10220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:02.828483 10220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:02.833895 10220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:02.835004 10220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:02.836046 10220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:02.846436 10226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:02.846451 10225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:02.846609 10220 server_base.cc:1034] running on GCE node
W20260620 07:12:02.846864 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:02.847388 10220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:02.848184 10220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:02.849407 10220 hybrid_clock.cc:648] HybridClock initialized: now 1781939522849392 us; error 31 us; skew 500 ppm
Jun 20 07:12:02 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:02.852319 10220 init.cc:377] Logged in from keytab as oryx/127.3.245.1@KRBTEST.COM (short username oryx)
I20260620 07:12:02.853299 10220 webserver.cc:460] Webserver started at http://127.3.245.1:35301/ using document root <none> and password file <none>
I20260620 07:12:02.853911 10220 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:02.854012 10220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:02.854254 10220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:02.856407 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "bdf3bb64790a41ab96a65bd641703bdc"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "42a160c9ed47def0407580c91a197b05"
server_key_iv: "61cf18910af7964c1cc0045415148bc0"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.857066 10220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "bdf3bb64790a41ab96a65bd641703bdc"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "42a160c9ed47def0407580c91a197b05"
server_key_iv: "61cf18910af7964c1cc0045415148bc0"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.861179 10220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:02.864017 10235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:02.865511 10220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:02.865618 10220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "bdf3bb64790a41ab96a65bd641703bdc"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "42a160c9ed47def0407580c91a197b05"
server_key_iv: "61cf18910af7964c1cc0045415148bc0"
server_key_version: "encryptionkey@0"
I20260620 07:12:02.865715 10220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:02.889156 10220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:02.892515 10220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:02.892769 10220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:02.893353 10220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:02.894172 10220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:02.894222 10220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:02.894265 10220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:02.894281 10220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:02.903784 10220 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:39229
I20260620 07:12:02.903812 10348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:39229 every 8 connection(s)
I20260620 07:12:02.904736 10220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:02.907493 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10220
I20260620 07:12:02.907639 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:02.907967 4052 external_mini_cluster.cc:1428] Setting key 688b4ae3c76df4da6a5faae33033512f
WARNING: no policy specified for oryx/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.2@KRBTEST.COM" created.
Jun 20 07:12:02 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:02.926314 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:02.906654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:38669 (local address 127.3.245.62:34251)
0620 07:12:02.906804 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:02.906807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:02.907735 (+ 928us) server_negotiation.cc:407] Connection header received
0620 07:12:02.909499 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:02.909503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:02.909557 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:02.909661 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:02.910865 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.911630 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.913157 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.913579 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.913760 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:02.921796 (+ 8036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:02.921803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:02.921809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:02.921851 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:02.923645 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:02.924436 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:02.924439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:02.924440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:02.924518 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:02.925012 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:02.925015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:02.925016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:02.925207 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:02.925294 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:02.926042 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 07:12:02.926155 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:12:02.927675 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:02.907029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:38669)
0620 07:12:02.907576 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:02.907629 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:02.909243 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:02.909854 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:02.909862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:02.910223 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:02.910705 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:02.910718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.911774 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.911777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:02.913024 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:02.913034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:02.913796 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:02.913799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:02.914000 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:02.917758 (+ 3758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:02.917782 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:02.921538 (+ 3756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:02.923797 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:02.923808 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:02.923831 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:02.924258 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:02.924651 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:02.924657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:02.924662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:02.924892 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:02.925323 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:02.925333 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:02.925750 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 07:12:02.925998 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 07:12:02.926433 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260620 07:12:02.929097 10349 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:02.929374 10349 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:02.929868 10349 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
I20260620 07:12:02.931464 10158 ts_manager.cc:194] Registered new tserver with Master: bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1:39229)
I20260620 07:12:02.932668 10158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:38669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
I20260620 07:12:02.939692 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:03.053416 10354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:03.053735 10354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:03.053784 10354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:03.058130 10354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:03.058213 10354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:03.058280 10354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:03.064136 10354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:03.065295 10354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:03.066372 10354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:03.077075 10360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.077075 10362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.077075 10359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.077626 10354 server_base.cc:1034] running on GCE node
I20260620 07:12:03.078049 10354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:03.078687 10354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:03.079872 10354 hybrid_clock.cc:648] HybridClock initialized: now 1781939523079839 us; error 46 us; skew 500 ppm
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:03.083020 10354 init.cc:377] Logged in from keytab as oryx/127.3.245.2@KRBTEST.COM (short username oryx)
I20260620 07:12:03.083933 10354 webserver.cc:460] Webserver started at http://127.3.245.2:35941/ using document root <none> and password file <none>
I20260620 07:12:03.084491 10354 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.084551 10354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:03.084794 10354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:03.086974 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "66148eca92d744c4b67f998ff067f505"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "fa2633556ca8239a43eb5e5887dee86d"
server_key_iv: "29f83803c3fe15bc533882ff26518453"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.087524 10354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "66148eca92d744c4b67f998ff067f505"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "fa2633556ca8239a43eb5e5887dee86d"
server_key_iv: "29f83803c3fe15bc533882ff26518453"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.091476 10354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:12:03.094125 10369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.095590 10354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:03.095752 10354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "66148eca92d744c4b67f998ff067f505"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "fa2633556ca8239a43eb5e5887dee86d"
server_key_iv: "29f83803c3fe15bc533882ff26518453"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.095878 10354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.115115 10354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:03.118468 10354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:03.118680 10354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:03.119290 10354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:03.120114 10354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:03.120189 10354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.120261 10354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:03.120314 10354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.129602 10354 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:39773
I20260620 07:12:03.129618 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:39773 every 8 connection(s)
I20260620 07:12:03.130592 10354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:03.137212 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10354
I20260620 07:12:03.137342 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:03.137629 4052 external_mini_cluster.cc:1428] Setting key d00c197f468209b069c17472adf4c247
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:03.146652 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.132597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:44653 (local address 127.3.245.62:34251)
0620 07:12:03.132778 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.132783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.133730 (+ 947us) server_negotiation.cc:407] Connection header received
0620 07:12:03.135271 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.135275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.135325 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.135405 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.136943 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.137782 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.139494 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.139880 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.140009 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.142909 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.142913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.142916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.142949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.144475 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.145060 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.145064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.145065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.145142 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.145494 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.145498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.145500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.145693 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.145809 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.146349 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.146481 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:12:03.147501 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.133085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:44653)
0620 07:12:03.133560 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.133606 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.135008 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.135615 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.135622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.136024 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.136592 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.136675 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.137927 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.137930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.139352 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.139363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.140046 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.140049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.140199 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.140969 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.140986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.142715 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.144602 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.144635 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.144656 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.144941 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.145241 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.145244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.145246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.145392 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.145811 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.145821 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.146094 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.146282 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.146655 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:12:03.148880 10483 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:03.149180 10483 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:03.149814 10483 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.3@KRBTEST.COM" created.
I20260620 07:12:03.151161 10158 ts_manager.cc:194] Registered new tserver with Master: 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:03.152022 10158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:44653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
I20260620 07:12:03.167941 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:03.287597 10488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:03.287938 10488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:03.288053 10488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:03.292253 10488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:03.292364 10488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:03.292459 10488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:03.297989 10488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:03.299216 10488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:03.300386 10488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:03.310844 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.310870 10494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.310855 10493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.311142 10488 server_base.cc:1034] running on GCE node
I20260620 07:12:03.311678 10488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:03.312439 10488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:03.313629 10488 hybrid_clock.cc:648] HybridClock initialized: now 1781939523313616 us; error 60 us; skew 500 ppm
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:03.316800 10488 init.cc:377] Logged in from keytab as oryx/127.3.245.3@KRBTEST.COM (short username oryx)
I20260620 07:12:03.317854 10488 webserver.cc:460] Webserver started at http://127.3.245.3:46019/ using document root <none> and password file <none>
I20260620 07:12:03.318431 10488 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.318518 10488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:03.318733 10488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:03.320881 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "970f2ebd3ae14132a5e63ccb53171632"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "f8258e1bdbbcc414c13f8390b0fe29df"
server_key_iv: "ebd474269b892e8aa0e2b9dac56bc85a"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.321446 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "970f2ebd3ae14132a5e63ccb53171632"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "f8258e1bdbbcc414c13f8390b0fe29df"
server_key_iv: "ebd474269b892e8aa0e2b9dac56bc85a"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.325517 10488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:12:03.328286 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.329788 10488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:12:03.329941 10488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "970f2ebd3ae14132a5e63ccb53171632"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "f8258e1bdbbcc414c13f8390b0fe29df"
server_key_iv: "ebd474269b892e8aa0e2b9dac56bc85a"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.330077 10488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.350435 10488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:03.354036 10488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:03.354264 10488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:03.354907 10488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:03.355767 10488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:03.355815 10488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.355880 10488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:03.355911 10488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.365311 10488 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:42611
I20260620 07:12:03.365370 10616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:42611 every 8 connection(s)
I20260620 07:12:03.366283 10488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:03.368093 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10488
I20260620 07:12:03.368297 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:03.368677 4052 external_mini_cluster.cc:1428] Setting key d20fa431f196ee3eeb15a9ba9ad403f5
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:03.381372 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.368254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:53663 (local address 127.3.245.62:34251)
0620 07:12:03.368421 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.368426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.370011 (+ 1585us) server_negotiation.cc:407] Connection header received
0620 07:12:03.371136 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.371139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.371202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.371313 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.372560 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.373206 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.374473 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.374719 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.374848 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.377521 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.377525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.377527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.377553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.378901 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.379564 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.379569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.379573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.379688 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.380081 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.380086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.380088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.380320 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.380444 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.381046 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.381188 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:12:03.382509 10619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.368740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:53663)
0620 07:12:03.369839 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.369926 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.370905 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.371499 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.371507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.371894 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.372409 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.372424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.373346 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.373350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.374339 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.374348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.374888 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.374893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.375071 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.375668 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.375692 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.377328 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.379030 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.379037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.379055 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.379376 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.379807 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.379811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.379814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.379970 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.380440 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.380448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.380787 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.381002 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.381376 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":638,"threads_started":1}
I20260620 07:12:03.383677 10617 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:03.383924 10617 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:03.384406 10617 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
I20260620 07:12:03.385432 10158 ts_manager.cc:194] Registered new tserver with Master: 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:03.386013 10158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:53663
I20260620 07:12:03.395720 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.245.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.61.keytab.
Entry for principal oryx/127.3.245.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.61.keytab.
I20260620 07:12:03.424856 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:34469
--webserver_interface=127.3.245.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.61.keytab
--principal=oryx/127.3.245.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34251,127.3.245.61:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:03.541429 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:03.541759 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:03.541846 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:03.546025 10622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:03.546113 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:03.546152 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:03.546211 10622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:03.546247 10622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:03.551770 10622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34251,127.3.245.61:34469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.61.keytab
--principal=oryx/127.3.245.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.245.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
W20260620 07:12:03.552115 10622 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.245.62:34251,127.3.245.61:34469'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 07:12:03.553084 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:03.554109 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:03.562906 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.563095 10622 server_base.cc:1034] running on GCE node
W20260620 07:12:03.562920 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.562999 10630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.563867 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:03.565256 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:03.566426 10622 hybrid_clock.cc:648] HybridClock initialized: now 1781939523566388 us; error 58 us; skew 500 ppm
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:03.569857 10622 init.cc:377] Logged in from keytab as oryx/127.3.245.61@KRBTEST.COM (short username oryx)
I20260620 07:12:03.571018 10622 webserver.cc:460] Webserver started at http://127.3.245.61:36651/ using document root <none> and password file <none>
I20260620 07:12:03.571638 10622 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.571734 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:03.571975 10622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:03.574288 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/instance:
uuid: "be0a80ea8be443c2963696a562eda479"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "51060f554b7fbd130ef27109ec9d3eb7"
server_key_iv: "fe4057a15673de0ef1b5efd454e1d3d8"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.574894 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal/instance:
uuid: "be0a80ea8be443c2963696a562eda479"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "51060f554b7fbd130ef27109ec9d3eb7"
server_key_iv: "fe4057a15673de0ef1b5efd454e1d3d8"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.578868 10622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:12:03.581687 10637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.583249 10622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:03.583388 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "be0a80ea8be443c2963696a562eda479"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "51060f554b7fbd130ef27109ec9d3eb7"
server_key_iv: "fe4057a15673de0ef1b5efd454e1d3d8"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.583513 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.605332 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:03.609170 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:03.609397 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:03.616963 10622 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.61:34469
I20260620 07:12:03.617334 10689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.61:34469 every 8 connection(s)
I20260620 07:12:03.618064 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
I20260620 07:12:03.620994 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:03.621965 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10622
I20260620 07:12:03.625593 10690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.61@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:03.637617 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.627396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.3.245.62:34251)
0620 07:12:03.627549 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.627554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.627921 (+ 367us) server_negotiation.cc:407] Connection header received
0620 07:12:03.628054 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.628060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.628117 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.628189 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.629069 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.629841 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.631194 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.631413 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.631501 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.634146 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.634150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.634152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.634181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.635765 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.636216 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.636218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.636220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.636315 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.636710 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.636713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.636715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.636975 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.637055 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.637406 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.637499 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:12:03.638127 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.627306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49126)
0620 07:12:03.627743 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.627812 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.627953 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.628273 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.628279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.628781 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.628941 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.628951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.629991 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.629995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.631047 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.631060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.631571 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.631576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.631712 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.632336 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.632353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.633992 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.635860 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.635866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.635873 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.636122 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.636413 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.636416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.636418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.636513 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.637086 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.637092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.637201 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.637353 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.637520 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":3,"thread_start_us":163,"threads_started":1}
I20260620 07:12:03.638127 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.623605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.3.245.61:34469)
0620 07:12:03.624054 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.624069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.624104 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:12:03.624869 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.624888 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.625231 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.625566 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.626415 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.627690 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.628982 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.629801 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.629991 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.633025 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.633031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.633042 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.633064 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.634748 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.635256 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.635261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.635266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.635372 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.635669 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.635671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.635672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.636219 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.636423 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.636726 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.637075 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":305,"thread_start_us":137,"threads_started":1}
I20260620 07:12:03.639616 10690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.61@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
I20260620 07:12:03.647277 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.640178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39464)
0620 07:12:03.640283 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.640308 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.640384 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.640593 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.640595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.640787 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.640862 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.640867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.641887 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.641890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.642392 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.642400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.642812 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.642815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.642898 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.643335 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.643345 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.644603 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.646046 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.646049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.646050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.646233 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.646472 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.646474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.646475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.646544 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.646861 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.646863 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.646935 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.647014 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.647107 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":18}
I20260620 07:12:03.647285 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.640247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.3.245.61:34469)
0620 07:12:03.640348 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.640351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.640363 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:12:03.640466 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.640468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.640508 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.640577 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.640959 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.641771 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.642485 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.642692 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.642859 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.644780 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.644783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.644785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.644805 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.645952 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.646327 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.646329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.646331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.646397 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.646615 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.646617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.646618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.646773 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.646858 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.647073 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.647158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:12:03.650241 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Bootstrap starting.
I20260620 07:12:03.652597 10690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:03.653388 10690 log.cc:824] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:03.655467 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: No bootstrap required, opened a new log
I20260620 07:12:03.658646 10690 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } }
I20260620 07:12:03.658885 10690 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:03.658974 10690 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0a80ea8be443c2963696a562eda479, State: Initialized, Role: FOLLOWER
I20260620 07:12:03.659426 10690 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } }
I20260620 07:12:03.659936 10696 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } } }
I20260620 07:12:03.660048 10696 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:03.660709 10690 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:12:03.665608 10707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:03.665728 10707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:03.667965 10622 master_runner.cc:428] Detected that this master be0a80ea8be443c2963696a562eda479 is joining an existing cluster
I20260620 07:12:03.668026 10622 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.245.61:34469
I20260620 07:12:03.679391 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.670221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39474)
0620 07:12:03.670582 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.670595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.670681 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.670904 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.670907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.671213 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.671361 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.671369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.672804 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.672807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.673455 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.673464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.674364 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.674368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.674491 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.674982 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.674995 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.675414 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.677665 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.677669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.677671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.678022 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.678311 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.678315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.678386 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.678809 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.678812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.678906 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.679013 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.679159 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":25,"spinlock_wait_cycles":1280,"thread_start_us":100,"threads_started":1}
I20260620 07:12:03.679577 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.670241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.3.245.62:34251)
0620 07:12:03.670372 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.670377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.670533 (+ 156us) server_negotiation.cc:407] Connection header received
0620 07:12:03.670725 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.670730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.670786 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.670842 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.671445 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.672237 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.673540 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.673825 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.673956 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.675782 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.675789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.675795 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.675832 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.677598 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678135 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.678139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.678142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.678236 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678593 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.678596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.678598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.678843 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.678915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.679287 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.679406 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 07:12:03.679437 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.670564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.3.245.61:34469)
0620 07:12:03.670668 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.670673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.670686 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:12:03.670753 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.670756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.670797 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.670869 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.671488 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.672243 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.673612 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.673925 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.674022 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.675554 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.675558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.675561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.675601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.677538 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678125 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.678129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.678130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.678217 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678471 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.678474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.678477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.678708 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.678810 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.679078 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.679239 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":4864}
I20260620 07:12:03.680104 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.670170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49132)
0620 07:12:03.670451 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.670467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.670607 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.670993 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.670995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.671207 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.671329 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.671335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.672355 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.672359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.673409 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.673420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.673972 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.673975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.674074 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.675129 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.675144 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.675620 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.677719 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.677722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.677724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.678026 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.678351 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.678355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.678357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.678473 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.678976 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.678978 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.679082 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.679749 (+ 667us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.679901 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":28,"spinlock_wait_cycles":10240,"thread_start_us":122,"threads_started":1}
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.62@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
I20260620 07:12:03.694679 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.684526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.3.245.61:34469)
0620 07:12:03.684704 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.684709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.684994 (+ 285us) server_negotiation.cc:407] Connection header received
0620 07:12:03.685215 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.685218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.685278 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.685368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.686050 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.686617 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.687583 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.687839 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.687955 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.690628 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.690635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.690637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.690665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.692572 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.693161 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.693168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.693173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.693281 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.693594 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.693598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.693599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.693821 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.693917 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.694393 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.694516 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:12:03.695020 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.684481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39476)
0620 07:12:03.684873 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.684915 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.685078 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.685405 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.685412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.685800 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.685927 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.685934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.686736 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.686740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.687457 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.687466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.687957 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.687961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.688087 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.688696 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.688709 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.690478 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.692716 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.692726 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.692733 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.693048 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.693386 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.693389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.693391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.693487 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.693951 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.693962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.694137 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.694322 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.694582 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":137,"threads_started":1}
I20260620 07:12:03.696513 10158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.245.61:34469: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "be0a80ea8be443c2963696a562eda479"
member_type: NON_VOTER
last_known_addr {
host: "127.3.245.61"
port: 34469
}
attrs {
promote: true
}
}
dest_uuid: "7563d4cea48b47cdab79bde300cbf61d"
cas_config_opid_index: -1
I20260620 07:12:03.697224 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } }
I20260620 07:12:03.699129 10719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER be0a80ea8be443c2963696a562eda479 (127.3.245.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } }
I20260620 07:12:03.699911 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:03.700822 10719 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.245.61:34469
I20260620 07:12:03.701226 10622 master.cc:537] Master@127.3.245.61:34469 shutting down...
I20260620 07:12:03.701290 10664 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
WARNING: no policy specified for oryx/127.3.245.60@KRBTEST.COM; defaulting to no policy
I20260620 07:12:03.704663 10719 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:03.704921 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } }
I20260620 07:12:03.705030 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:03.706130 10622 raft_consensus.cc:2229] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:03.706457 10622 raft_consensus.cc:2258] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:03.706601 10622 tablet_replica.cc:330] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.60@KRBTEST.COM" created.
W20260620 07:12:03.707363 10142 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 07:12:03.710390 10142 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d -> Peer be0a80ea8be443c2963696a562eda479 (127.3.245.61:34469): Couldn't send request to peer be0a80ea8be443c2963696a562eda479. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:12:03.722323 10622 master.cc:552] Master@127.3.245.61:34469 shutdown complete.
I20260620 07:12:03.722404 10622 master_runner.cc:305] Clearing existing system tablet
I20260620 07:12:03.722642 10622 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.722714 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.60.keytab.
Entry for principal oryx/127.3.245.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.60.keytab.
I20260620 07:12:03.723884 10622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 07:12:03.724768 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:34457
--webserver_interface=127.3.245.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.60.keytab
--principal=oryx/127.3.245.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:03.724862 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.725239 10622 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.725364 10622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:03.725397 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "be0a80ea8be443c2963696a562eda479"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "51060f554b7fbd130ef27109ec9d3eb7"
server_key_iv: "fe4057a15673de0ef1b5efd454e1d3d8"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.727612 10622 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:03.730118 10622 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:12:03.730211 10622 log.cc:1196] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 07:12:03.730459 10622 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Deleting consensus metadata
I20260620 07:12:03.730624 10622 master_runner.cc:315] Copying system tablet from 127.3.245.62:34251
I20260620 07:12:03.731793 10622 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: tablet copy: Beginning tablet copy session from remote peer at address 127.3.245.62:34251
I20260620 07:12:03.742903 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.732390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49146)
0620 07:12:03.734356 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.734376 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.734543 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.734893 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.734899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.735147 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.735367 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.735379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.736424 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.736427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.737427 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.737438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.738046 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.738050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.738149 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.738769 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.738784 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.739221 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.741224 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.741227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.741230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.741497 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.741794 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.741800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.741805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.741905 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.742341 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.742344 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.742450 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.742559 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.742703 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1855,"thread_start_us":106,"threads_started":1}
I20260620 07:12:03.742918 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.732473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.3.245.62:34251)
0620 07:12:03.732599 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.732604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.734496 (+ 1892us) server_negotiation.cc:407] Connection header received
0620 07:12:03.734693 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.734697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.734756 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.734843 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.735489 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.736305 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.737567 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.737924 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.738027 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.739366 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.739372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.739375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.739402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.741109 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.741598 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.741602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.741604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.741698 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.742007 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.742011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.742013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.742230 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.742331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.742628 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.742756 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:12:03.743923 10178 tablet_copy_service.cc:140] P 7563d4cea48b47cdab79bde300cbf61d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer be0a80ea8be443c2963696a562eda479 ({username='oryx', principal='oryx/127.3.245.61@KRBTEST.COM'} at 127.0.0.1:49146)
I20260620 07:12:03.744122 10178 tablet_copy_service.cc:161] P 7563d4cea48b47cdab79bde300cbf61d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer be0a80ea8be443c2963696a562eda479 at {username='oryx', principal='oryx/127.3.245.61@KRBTEST.COM'} at 127.0.0.1:49146: session id = be0a80ea8be443c2963696a562eda479-00000000000000000000000000000000
I20260620 07:12:03.745970 10178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:12:03.747733 10622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:12:03.750193 10622 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: tablet copy: Starting download of 0 data blocks...
I20260620 07:12:03.750453 10622 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: tablet copy: Starting download of 1 WAL segments...
I20260620 07:12:03.752332 10622 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:12:03.754341 10178 tablet_copy_service.cc:342] P 7563d4cea48b47cdab79bde300cbf61d: Request end of tablet copy session be0a80ea8be443c2963696a562eda479-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.245.61@KRBTEST.COM'} at 127.0.0.1:49146
I20260620 07:12:03.754494 10178 tablet_copy_service.cc:433] P 7563d4cea48b47cdab79bde300cbf61d: ending tablet copy session be0a80ea8be443c2963696a562eda479-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer be0a80ea8be443c2963696a562eda479
I20260620 07:12:03.762868 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:03.764789 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.764806 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.765293 10622 server_base.cc:1034] running on GCE node
W20260620 07:12:03.768452 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.768726 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:03.769089 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:03.776469 10622 hybrid_clock.cc:648] HybridClock initialized: now 1781939523776420 us; error 70 us; skew 500 ppm
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:03.778304 10622 init.cc:377] Logged in from keytab as oryx/127.3.245.61@KRBTEST.COM (short username oryx)
I20260620 07:12:03.778831 10622 webserver.cc:460] Webserver started at http://127.3.245.61:33151/ using document root <none> and password file <none>
I20260620 07:12:03.779032 10622 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.779074 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:03.780464 10622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:03.781522 10743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.781780 10622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:03.781855 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "be0a80ea8be443c2963696a562eda479"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "51060f554b7fbd130ef27109ec9d3eb7"
server_key_iv: "fe4057a15673de0ef1b5efd454e1d3d8"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.781935 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.806818 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:03.809281 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:03.814507 10622 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.61:34469
I20260620 07:12:03.814541 10795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.61:34469 every 8 connection(s)
I20260620 07:12:03.815191 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
I20260620 07:12:03.817340 10796 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:03.818539 10796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Bootstrap starting.
I20260620 07:12:03.831526 10796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:03.831920 10796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: Bootstrap complete.
I20260620 07:12:03.832597 10796 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } }
I20260620 07:12:03.832780 10796 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be0a80ea8be443c2963696a562eda479, State: Initialized, Role: LEARNER
I20260620 07:12:03.832929 10796 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } }
I20260620 07:12:03.833359 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } }
I20260620 07:12:03.833467 10796 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:03.833596 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:12:03.837347 10810 catalog_manager.cc:1259] Loaded cluster ID: 5cb1b2eb4c974bd8a8f17109f2f80fc4
I20260620 07:12:03.837410 10810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: loading cluster ID for follower catalog manager: success
I20260620 07:12:03.839718 10810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: acquiring CA information for follower catalog manager: success
I20260620 07:12:03.841163 10810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:12:03.865593 10726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:03.865978 10726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:03.866080 10726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:03.870970 10726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:03.871059 10726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:03.871131 10726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:03.871192 10726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:03.871232 10726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:03.877270 10726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.60.keytab
--principal=oryx/127.3.245.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.245.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:03.878461 10726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:03.879529 10726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:03.888778 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.888792 10726 server_base.cc:1034] running on GCE node
W20260620 07:12:03.888787 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:03.888813 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:03.889650 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:03.890848 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:03.892037 10726 hybrid_clock.cc:648] HybridClock initialized: now 1781939523892019 us; error 41 us; skew 500 ppm
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:03.895188 10726 init.cc:377] Logged in from keytab as oryx/127.3.245.60@KRBTEST.COM (short username oryx)
I20260620 07:12:03.896201 10726 webserver.cc:460] Webserver started at http://127.3.245.60:40917/ using document root <none> and password file <none>
I20260620 07:12:03.896823 10726 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:03.896893 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:03.897068 10726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:03.899211 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/instance:
uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "0b0cb36bd75d2c6538521deebcff1af9"
server_key_iv: "acddacd50c3613cc26c4c95b2446f791"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.899858 10726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal/instance:
uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "0b0cb36bd75d2c6538521deebcff1af9"
server_key_iv: "acddacd50c3613cc26c4c95b2446f791"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.903951 10726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:12:03.906934 10825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:03.908530 10726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:03.908708 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "0b0cb36bd75d2c6538521deebcff1af9"
server_key_iv: "acddacd50c3613cc26c4c95b2446f791"
server_key_version: "encryptionkey@0"
I20260620 07:12:03.908860 10726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:03.935475 10349 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
I20260620 07:12:03.941381 10726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:03.945021 10726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:03.945207 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:03.954125 10726 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.60:34457
I20260620 07:12:03.954437 10877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.60:34457 every 8 connection(s)
I20260620 07:12:03.955217 10726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
I20260620 07:12:03.958520 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:03.962513 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10726
I20260620 07:12:03.962510 10878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.60@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
I20260620 07:12:03.975840 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.964872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.3.245.62:34251)
0620 07:12:03.964988 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.964992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.965190 (+ 198us) server_negotiation.cc:407] Connection header received
0620 07:12:03.965832 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.965835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.965877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.965947 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.966879 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.967407 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.968564 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.968874 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.968954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.971684 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.971688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.971692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.971726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.973409 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.974053 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.974056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.974057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.974128 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.974503 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.974509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.974514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.974752 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.974851 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.975510 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.975648 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 07:12:03.976958 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.3.245.60:34457)
0620 07:12:03.964495 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.964510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.964581 (+ 71us) server_negotiation.cc:407] Connection header received
0620 07:12:03.965624 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.965647 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.965882 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.966046 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.966806 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.967840 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.968764 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.969229 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.969390 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.972111 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.972117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.972128 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.972154 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.974149 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.974578 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.974583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.974584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.974664 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.974973 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.974977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.974978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.975327 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.975527 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.975811 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.975938 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":132,"threads_started":1}
I20260620 07:12:03.976958 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.964690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49160)
0620 07:12:03.965038 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.965089 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.965624 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.965997 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.966003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.966435 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.966696 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.966709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.967535 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.967539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.968457 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.968466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.969010 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.969014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.969168 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.969862 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.969878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.971511 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.973523 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.973531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.973543 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.973950 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.974217 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.974224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.974228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.974396 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.974854 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.974859 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.975201 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.975429 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.975800 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260620 07:12:03.979022 10878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.60@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
I20260620 07:12:03.987805 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.979643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39482)
0620 07:12:03.979740 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.979754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.979826 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.980217 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.980220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.980401 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.980489 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.980494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.981428 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.981431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.981976 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.981983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.982480 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.982484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.982587 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.983071 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.983092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.984345 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.986425 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.986428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.986429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.986624 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.986883 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.986885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.986886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.986956 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.987349 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.987351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.987442 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.987557 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.987676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 07:12:03.987883 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.979763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.3.245.61:34469)
0620 07:12:03.980021 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.980024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.980035 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:12:03.980068 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.980070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.980106 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.980205 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.980604 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.981318 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.982090 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.982373 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.982482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.984481 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.984484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.984487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.984511 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.986317 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.986712 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.986714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.986716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.986804 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.987031 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.987033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.987034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.987228 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.987311 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.987622 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.987722 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260620 07:12:03.988821 10878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:12:03 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.60@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
I20260620 07:12:03.997323 10881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.989371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41650)
0620 07:12:03.989497 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:03.989511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:03.989587 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:03.989819 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:03.989822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:03.989967 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:03.990048 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.990053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.990925 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.990928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.991532 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:03.991540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.992390 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.992393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:03.992453 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.992916 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:03.992927 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:03.994191 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:03.995850 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.995852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.995853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.996046 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.996277 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:03.996279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:03.996280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:03.996357 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:03.996912 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:03.996915 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:03.996992 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 07:12:03.997078 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:12:03.997202 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:12:03.997465 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:03.989474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.3.245.60:34457)
0620 07:12:03.989580 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:03.989583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:03.989595 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:12:03.989662 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:03.989664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:03.989696 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:03.989764 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:03.990238 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.990810 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.991641 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:03.992250 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:03.992340 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:03.994309 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:03.994312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:03.994314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:03.994334 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:03.995706 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.996130 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.996132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.996133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.996199 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:03.996479 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:03.996482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:03.996484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:03.996791 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:03.996885 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:03.997148 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 07:12:03.997265 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":41}
I20260620 07:12:04.000442 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Bootstrap starting.
I20260620 07:12:04.002777 10878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:04.003485 10878 log.cc:824] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:04.005437 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: No bootstrap required, opened a new log
I20260620 07:12:04.008751 10878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } }
I20260620 07:12:04.009053 10878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:04.009102 10878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec43a8c6dec45a2afc7868958cd1a56, State: Initialized, Role: FOLLOWER
I20260620 07:12:04.009563 10878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } }
I20260620 07:12:04.010082 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } } }
I20260620 07:12:04.010222 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:04.010864 10878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:12:04.015681 10895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:04.015776 10895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:04.019364 10726 master_runner.cc:428] Detected that this master 3ec43a8c6dec45a2afc7868958cd1a56 is joining an existing cluster
I20260620 07:12:04.019443 10726 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.245.60:34457
I20260620 07:12:04.030594 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10220
I20260620 07:12:04.031857 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.3.245.60:34457)
0620 07:12:04.022667 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.022672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.023324 (+ 652us) server_negotiation.cc:407] Connection header received
0620 07:12:04.023402 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.023405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.023464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.023550 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.024243 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025057 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.026226 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.026573 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.026656 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.028122 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.028125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.028128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.028150 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.029862 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.030422 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.030426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.030430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.030535 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.030858 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.030864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.030868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.031110 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.031191 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.031465 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.031680 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":127}
I20260620 07:12:04.031865 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41654)
0620 07:12:04.023188 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.023202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.023310 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.023696 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.023699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.023951 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.024117 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.024126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025182 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.025185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.026103 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.026114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.026834 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.026837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.026921 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.027478 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.027493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.027991 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.029977 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.029980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.029982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.030210 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.030645 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.030648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.030650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.030757 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.031241 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.031243 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.031309 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.031419 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.031603 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260620 07:12:04.040300 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:39229
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:34251
--webserver_port=35301
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:04.067303 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49174)
0620 07:12:04.023565 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.023583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.023733 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.024142 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.024147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.024357 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.024487 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.024495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025614 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.025617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.026757 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.026768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.027568 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.027572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.027665 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.028219 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.028236 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.061365 (+ 33129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.064545 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.064550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.064552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.065035 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.065720 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.065731 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.065824 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.066347 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.066350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.066759 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.066883 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.067057 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":387,"spinlock_wait_cycles":1920,"thread_start_us":82,"threads_started":1}
I20260620 07:12:04.067345 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.3.245.62:34251)
0620 07:12:04.022790 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.022797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.023666 (+ 869us) server_negotiation.cc:407] Connection header received
0620 07:12:04.023866 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.023870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.023924 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.023987 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.024684 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025432 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.026922 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.027282 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.027376 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.061617 (+ 34241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.061622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.061626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.061664 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.064392 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065203 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.065208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.065210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.065346 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065945 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.065949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.065950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.066193 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.066282 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.067031 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.067170 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260620 07:12:04.067960 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.3.245.61:34469)
0620 07:12:04.023141 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.023147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.023165 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:04.023240 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.023247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.023310 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.023394 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.024266 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025091 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.026922 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.027655 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.027742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.062288 (+ 34546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.062294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.062298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.062335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.064363 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065208 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.065212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.065214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.065505 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065925 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.065929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.065931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.066189 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.066275 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.067646 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.067804 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 07:12:04.068689 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.022406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39494)
0620 07:12:04.022912 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.022927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.023023 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.023533 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.023537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.023865 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.024034 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.024122 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.025268 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.025272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.026757 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.026768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.028248 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.028252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.028365 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.061373 (+ 33008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.061393 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.062062 (+ 669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.064646 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.064650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.064652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.065028 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.065658 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.065665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.065683 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.065796 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.066349 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.066352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.066530 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.068172 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.068340 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":108,"thread_start_us":153,"threads_started":1}
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.62@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
I20260620 07:12:04.081499 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.071610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41668)
0620 07:12:04.071784 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.071798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.071871 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.072179 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.072182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.072381 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.072498 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.072504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.073565 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.073568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.074432 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.074443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.075069 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.075073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.075175 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.075725 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.075741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.077475 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.079655 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.079658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.079661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.079933 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.080267 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.080270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.080272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.080364 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.080895 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.080898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.081005 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.081115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.081278 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 07:12:04.082131 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.071689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.3.245.60:34457)
0620 07:12:04.071806 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.071812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.071830 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:04.072000 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.072003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.072057 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.072149 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.072647 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.073449 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.074559 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.074938 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.075048 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.077669 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.077674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.077677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.077706 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.079496 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.080058 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.080061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.080064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.080156 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.080469 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.080472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.080474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.080746 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.080845 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.081517 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.081975 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:12:04.083153 10158 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.245.60:34457: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
member_type: NON_VOTER
last_known_addr {
host: "127.3.245.60"
port: 34457
}
attrs {
promote: true
}
}
dest_uuid: "7563d4cea48b47cdab79bde300cbf61d"
cas_config_opid_index: 5
I20260620 07:12:04.083773 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } }
I20260620 07:12:04.085194 10719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.086670 10852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:04.086889 10719 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.245.60:34457
I20260620 07:12:04.087075 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.087170 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.087792 10726 master.cc:537] Master@127.3.245.60:34457 shutting down...
I20260620 07:12:04.088076 10852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:12:04.089272 10720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:12:04.091384 10145 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d -> Peer 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60:34457): Couldn't send request to peer 3ec43a8c6dec45a2afc7868958cd1a56. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 07:12:04.095122 10726 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:04.095448 10726 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:04.095569 10726 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: stopping tablet replica
I20260620 07:12:04.096632 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.085392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39510)
0620 07:12:04.085992 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.086007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.086075 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.092124 (+ 6049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.092128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.092147 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:04.092239 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.092245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.093419 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.093422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.094679 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.094690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.096163 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.096167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.096267 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.096317 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.096386 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.096467 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525}
I20260620 07:12:04.097070 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.088795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.3.245.61:34469)
0620 07:12:04.091643 (+ 2848us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.091647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.091663 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:12:04.091714 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.091717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.091794 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.091894 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:04.092444 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.093276 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.094837 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.095823 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.095909 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.096064 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.096877 (+ 813us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.096958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2737}
I20260620 07:12:04.097882 10762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 LEARNER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 07:12:04.098548 10720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 07:12:04.101017 10726 master.cc:552] Master@127.3.245.60:34457 shutdown complete.
I20260620 07:12:04.101069 10726 master_runner.cc:305] Clearing existing system tablet
I20260620 07:12:04.101330 10726 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:04.101076 10762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.101387 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:04.102672 10726 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 07:12:04.103255 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7563d4cea48b47cdab79bde300cbf61d. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.103360 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:12:04.103443 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: NON_VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: true } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.103479 10908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.103503 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:12:04.103664 10726 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:04.103783 10726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:04.103823 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "0b0cb36bd75d2c6538521deebcff1af9"
server_key_iv: "acddacd50c3613cc26c4c95b2446f791"
server_key_version: "encryptionkey@0"
I20260620 07:12:04.105193 10720 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: attempting to promote NON_VOTER be0a80ea8be443c2963696a562eda479 to VOTER
I20260620 07:12:04.105686 10720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } }
I20260620 07:12:04.106058 10726 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:04.107903 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.107123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41678)
0620 07:12:04.107709 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.107811 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496}
I20260620 07:12:04.108932 10762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 LEARNER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 07:12:04.109025 10726 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 07:12:04.109104 10726 log.cc:1196] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 07:12:04.109337 10726 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Deleting consensus metadata
I20260620 07:12:04.109503 10726 master_runner.cc:315] Copying system tablet from 127.3.245.62:34251
I20260620 07:12:04.109915 10718 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 07:12:04.110625 10726 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: tablet copy: Beginning tablet copy session from remote peer at address 127.3.245.62:34251
I20260620 07:12:04.111168 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.110793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41694)
0620 07:12:04.111022 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.111085 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260620 07:12:04.111971 10145 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d -> Peer 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60:34457): Couldn't send request to peer 3ec43a8c6dec45a2afc7868958cd1a56. Status: Network error: Client connection negotiation failed: client connection to 127.3.245.60:34457: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:12:04.112967 10719 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, be0a80ea8be443c2963696a562eda479 (127.3.245.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.113497 10762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, be0a80ea8be443c2963696a562eda479 (127.3.245.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.114499 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.114610 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.114727 10719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.114786 10719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.115345 10907 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.115478 10907 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:04.124747 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.111355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.3.245.62:34251)
0620 07:12:04.111504 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.111509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.115877 (+ 4368us) server_negotiation.cc:407] Connection header received
0620 07:12:04.116014 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.116017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.116069 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.116157 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.116813 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.117603 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.119016 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.119339 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.119469 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.120908 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.120912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.120915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.120950 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.122750 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.123282 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.123286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.123288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.123379 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.123698 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.123703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.123707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.123932 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.124029 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.124411 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.124547 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:12:04.124855 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.111607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49188)
0620 07:12:04.115770 (+ 4163us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.115789 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.115914 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.116200 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.116203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.116436 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.116551 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.116558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.117726 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.117737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.118836 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.118848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.119488 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.119492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.119590 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.120234 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.120251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.120759 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.122875 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.122879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.122882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.123161 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.123479 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.123483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.123485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.123583 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.124037 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.124040 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.124152 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.124261 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.124464 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":68,"threads_started":1}
I20260620 07:12:04.125610 10178 tablet_copy_service.cc:140] P 7563d4cea48b47cdab79bde300cbf61d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ec43a8c6dec45a2afc7868958cd1a56 ({username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:49188)
I20260620 07:12:04.125711 10178 tablet_copy_service.cc:161] P 7563d4cea48b47cdab79bde300cbf61d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ec43a8c6dec45a2afc7868958cd1a56 at {username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:49188: session id = 3ec43a8c6dec45a2afc7868958cd1a56-00000000000000000000000000000000
I20260620 07:12:04.126801 10178 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 07:12:04.128388 10726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 07:12:04.130908 10726 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: tablet copy: Starting download of 0 data blocks...
I20260620 07:12:04.131102 10726 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: tablet copy: Starting download of 1 WAL segments...
I20260620 07:12:04.135738 10726 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 07:12:04.138231 10178 tablet_copy_service.cc:342] P 7563d4cea48b47cdab79bde300cbf61d: Request end of tablet copy session 3ec43a8c6dec45a2afc7868958cd1a56-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:49188
I20260620 07:12:04.138350 10178 tablet_copy_service.cc:433] P 7563d4cea48b47cdab79bde300cbf61d: ending tablet copy session 3ec43a8c6dec45a2afc7868958cd1a56-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ec43a8c6dec45a2afc7868958cd1a56
I20260620 07:12:04.146034 10726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:04.147823 10915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.149935 10916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.150014 10918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:04.152788 10726 server_base.cc:1034] running on GCE node
I20260620 07:12:04.153090 10726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:04.153416 10726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:04.154574 10726 hybrid_clock.cc:648] HybridClock initialized: now 1781939524154563 us; error 34 us; skew 500 ppm
I20260620 07:12:04.155069 10483 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:04.156319 10726 init.cc:377] Logged in from keytab as oryx/127.3.245.60@KRBTEST.COM (short username oryx)
I20260620 07:12:04.156880 10726 webserver.cc:460] Webserver started at http://127.3.245.60:38861/ using document root <none> and password file <none>
I20260620 07:12:04.157070 10726 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:04.157112 10726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:04.158622 10726 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:04.159992 10925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.160312 10726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:04.160398 10726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "0b0cb36bd75d2c6538521deebcff1af9"
server_key_iv: "acddacd50c3613cc26c4c95b2446f791"
server_key_version: "encryptionkey@0"
I20260620 07:12:04.160490 10726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:04.184167 10726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:04.186698 10726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:04.191795 10726 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.60:34457
I20260620 07:12:04.191835 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.60:34457 every 8 connection(s)
I20260620 07:12:04.192646 10726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
I20260620 07:12:04.194630 10978 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:04.195803 10978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Bootstrap starting.
W20260620 07:12:04.207190 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:04.207608 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:04.207721 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:12:04.209048 10978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:04.209354 10978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Bootstrap complete.
I20260620 07:12:04.209954 10978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } }
I20260620 07:12:04.210095 10978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ec43a8c6dec45a2afc7868958cd1a56, State: Initialized, Role: LEARNER
I20260620 07:12:04.210193 10978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } }
I20260620 07:12:04.210551 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.210657 10978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:04.210774 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: LEARNER
I20260620 07:12:04.213708 10995 catalog_manager.cc:1259] Loaded cluster ID: 5cb1b2eb4c974bd8a8f17109f2f80fc4
I20260620 07:12:04.213848 10995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: loading cluster ID for follower catalog manager: success
W20260620 07:12:04.214645 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:04.214736 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:04.214848 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:04.216256 10995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: acquiring CA information for follower catalog manager: success
I20260620 07:12:04.217756 10995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:12:04.220484 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:39229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=35301
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:04.221535 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:04.222548 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:04.232720 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.232731 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.232667 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:04.232832 10904 server_base.cc:1034] running on GCE node
I20260620 07:12:04.233572 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:04.234294 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:04.235482 10904 hybrid_clock.cc:648] HybridClock initialized: now 1781939524235472 us; error 49 us; skew 500 ppm
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:04.238320 10904 init.cc:377] Logged in from keytab as oryx/127.3.245.1@KRBTEST.COM (short username oryx)
I20260620 07:12:04.239252 10904 webserver.cc:460] Webserver started at http://127.3.245.1:35301/ using document root <none> and password file <none>
I20260620 07:12:04.239811 10904 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:04.239913 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:04.243799 10904 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:04.245978 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.247247 10904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:04.247395 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "bdf3bb64790a41ab96a65bd641703bdc"
format_stamp: "Formatted at 2026-06-20 07:12:02 on dist-test-slave-ldfr"
server_key: "42a160c9ed47def0407580c91a197b05"
server_key_iv: "61cf18910af7964c1cc0045415148bc0"
server_key_version: "encryptionkey@0"
I20260620 07:12:04.247951 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:04.277029 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:04.280491 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:04.280726 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:04.281334 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:04.282269 10904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:04.282339 10904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.282406 10904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:04.282452 10904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.292394 10904 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:39229
I20260620 07:12:04.292424 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:39229 every 8 connection(s)
I20260620 07:12:04.293511 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:04.300135 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 10904
I20260620 07:12:04.300343 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10354
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
I20260620 07:12:04.308600 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:39773
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:34251
--webserver_port=35941
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:04.312533 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.295673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:58471 (local address 127.3.245.61:34469)
0620 07:12:04.295798 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.295803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.297385 (+ 1582us) server_negotiation.cc:407] Connection header received
0620 07:12:04.298239 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.298245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.298305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.298383 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.299917 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.300770 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.302646 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.302998 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.303077 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.307037 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.307042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.307046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.307080 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.309133 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.309880 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.309884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.309886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.310008 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.310948 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.310952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.310954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.311204 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.311320 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.312177 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.312314 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:12:04.312662 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.295424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48889 (local address 127.3.245.60:34457)
0620 07:12:04.296390 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.296395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.296821 (+ 426us) server_negotiation.cc:407] Connection header received
0620 07:12:04.298507 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.298511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.298561 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.298661 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.301400 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.302496 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.303557 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.304471 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.304557 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.307659 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.307666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.307670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.307704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.310540 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311280 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.311286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.311291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.311383 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311693 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.311697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.311699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.311914 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.311997 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.312351 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.312473 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":88,"threads_started":1}
I20260620 07:12:04.312794 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.296134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:60281 (local address 127.3.245.62:34251)
0620 07:12:04.296285 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.296289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.297146 (+ 857us) server_negotiation.cc:407] Connection header received
0620 07:12:04.298110 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.298114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.298180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.298263 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.299901 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.300650 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.302852 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.303290 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.303385 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.307968 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.307972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.307978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.308015 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.310411 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311128 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.311134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.311137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.311238 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311555 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.311560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.311564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.311802 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.311907 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.312370 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.312510 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:12:04.314411 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.295924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.3.245.1:58471)
0620 07:12:04.297285 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.297298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.297881 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.298467 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.298473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.299148 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.299749 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.299755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.301011 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.301017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.302508 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.302517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.303291 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.303294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.303489 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.304195 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.304218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.306856 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.309280 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.309285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.309300 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.309732 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.310685 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.310688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.310690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.310845 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.311404 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.311414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.311893 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.313807 (+ 1914us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.313943 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":267,"spinlock_wait_cycles":12544,"thread_start_us":118,"threads_started":1}
I20260620 07:12:04.314422 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.295923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.3.245.1:48889)
0620 07:12:04.296666 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.296721 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.297851 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.298693 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.298696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.299142 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.299746 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.299755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.302664 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.302667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.303431 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.303438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.304745 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.304748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.304842 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.305334 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.305355 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.307401 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.310865 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.310869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.310872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.311160 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.311489 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.311493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.311576 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.312055 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.312061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.312177 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.312825 (+ 648us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.313278 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":124,"spinlock_wait_cycles":46848,"thread_start_us":150,"threads_started":1}
I20260620 07:12:04.314734 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.295924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:60281)
0620 07:12:04.297000 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.297012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.297851 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.298464 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.298474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.299158 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.299739 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.299753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.301616 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.301622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.302709 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.302722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.303554 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.303560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.303700 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.304308 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.304329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.307780 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.310533 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.310538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.310541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.311007 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.311360 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.311365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.311368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.311451 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.312029 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.312034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.312170 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.313000 (+ 830us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.313379 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":218,"spinlock_wait_cycles":31744,"thread_start_us":128,"threads_started":1}
I20260620 07:12:04.315907 11121 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:04.315920 11123 heartbeater.cc:347] Connected to a master server at 127.3.245.61:34469
I20260620 07:12:04.315922 11122 heartbeater.cc:347] Connected to a master server at 127.3.245.60:34457
I20260620 07:12:04.316190 11121 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.316179 11123 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.316179 11122 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.316895 11123 heartbeater.cc:511] Master 127.3.245.61:34469 requested a full tablet report, sending...
I20260620 07:12:04.316895 11121 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
I20260620 07:12:04.317072 11122 heartbeater.cc:511] Master 127.3.245.60:34457 requested a full tablet report, sending...
I20260620 07:12:04.318313 10158 ts_manager.cc:194] Re-registered known tserver with Master: bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1:39229)
I20260620 07:12:04.318519 10752 ts_manager.cc:194] Registered new tserver with Master: bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1:39229)
I20260620 07:12:04.318519 10942 ts_manager.cc:194] Registered new tserver with Master: bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1:39229)
I20260620 07:12:04.319242 10158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:60281
I20260620 07:12:04.388090 10617 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
W20260620 07:12:04.427604 11129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:04.427922 11129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:04.428000 11129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:04.432071 11129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:04.432183 11129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:04.432282 11129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:04.438155 11129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:39773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=35941
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:04.439215 11129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:04.440250 11129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:04.450006 11134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.450030 11135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:04.450215 11129 server_base.cc:1034] running on GCE node
W20260620 07:12:04.450387 11137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:04.450767 11129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:04.451434 11129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:04.452688 11129 hybrid_clock.cc:648] HybridClock initialized: now 1781939524452667 us; error 29 us; skew 500 ppm
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:04.455904 11129 init.cc:377] Logged in from keytab as oryx/127.3.245.2@KRBTEST.COM (short username oryx)
I20260620 07:12:04.456878 11129 webserver.cc:460] Webserver started at http://127.3.245.2:35941/ using document root <none> and password file <none>
I20260620 07:12:04.457398 11129 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:04.457466 11129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:04.461524 11129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:04.463615 11144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.464948 11129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:12:04.465068 11129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "66148eca92d744c4b67f998ff067f505"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "fa2633556ca8239a43eb5e5887dee86d"
server_key_iv: "29f83803c3fe15bc533882ff26518453"
server_key_version: "encryptionkey@0"
I20260620 07:12:04.465608 11129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:04.479895 11129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:04.483256 11129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:04.483413 11129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:04.483940 11129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:04.484818 11129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:04.484867 11129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.484907 11129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:04.484922 11129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.493980 11129 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:39773
I20260620 07:12:04.494014 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:39773 every 8 connection(s)
I20260620 07:12:04.495005 11129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:04.496769 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11129
I20260620 07:12:04.496948 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10488
I20260620 07:12:04.505774 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:42611
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:34251
--webserver_port=46019
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
I20260620 07:12:04.513921 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.497082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:59947 (local address 127.3.245.62:34251)
0620 07:12:04.497398 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.497404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.500593 (+ 3189us) server_negotiation.cc:407] Connection header received
0620 07:12:04.500816 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.500819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.500872 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.500924 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.501663 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502481 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.504720 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505003 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505127 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.509301 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.509305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.509307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.509335 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.511185 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.511932 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.511938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.511943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.512040 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512512 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.512516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.512518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.512797 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.512883 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.513611 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.513745 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260620 07:12:04.513921 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.497471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:53533 (local address 127.3.245.61:34469)
0620 07:12:04.497615 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.497621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.500266 (+ 2645us) server_negotiation.cc:407] Connection header received
0620 07:12:04.500365 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.500371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.500431 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.500512 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.501648 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502537 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.504761 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505091 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505209 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.510095 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.510101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.510105 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.510141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.511899 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512428 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.512434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.512439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.512527 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512868 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.512872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.512873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.513079 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.513155 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.513615 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.513752 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:12:04.514900 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.497238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:49463 (local address 127.3.245.60:34457)
0620 07:12:04.498672 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.498677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.498696 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:12:04.499724 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.499729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.499795 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.499896 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.501655 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502481 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.504795 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505154 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505254 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.509075 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.509082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.509085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.509128 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.511667 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512409 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.512414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.512418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.512538 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512949 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.512954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.512958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.513234 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.513351 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.514594 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.514737 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329}
I20260620 07:12:04.515234 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.497615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.3.245.2:53533)
0620 07:12:04.500154 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.500168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.500274 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.500668 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.500674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.500917 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.501488 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.501507 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502689 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.502695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.504550 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.504565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505346 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.505552 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.506387 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.506415 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.509916 (+ 3501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.512012 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.512017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.512310 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.512667 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.512671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.512761 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.513206 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.513209 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.513304 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.514639 (+ 1335us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.514786 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449,"mutex_wait_us":488,"spinlock_wait_cycles":49024,"thread_start_us":154,"threads_started":1}
I20260620 07:12:04.515259 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.497616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.3.245.2:49463)
0620 07:12:04.498186 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.498245 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.499472 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.500095 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.500105 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.500693 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.501491 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.501502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502669 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.502672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.504715 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.504720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505363 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.505556 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.506659 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.506679 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.508776 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.511870 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.511877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.511881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.512264 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.512682 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.512693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.512805 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.513407 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.513414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.513554 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.513698 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.514114 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":644,"spinlock_wait_cycles":20992,"thread_start_us":146,"threads_started":1}
I20260620 07:12:04.515484 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.498680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:59947)
0620 07:12:04.500507 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.500521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.500699 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.501056 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.501059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.501243 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.501481 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.501502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.502669 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.502672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.504564 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.504576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.505177 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.505181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.505395 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.506506 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.506524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.509026 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.511404 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.511413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.511437 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.511801 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.512154 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.512157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.512158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.512356 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.512949 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.512955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.513309 (+ 354us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.514068 (+ 759us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.514243 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"mutex_wait_us":492,"spinlock_wait_cycles":14336,"thread_start_us":101,"threads_started":1}
I20260620 07:12:04.516731 11260 heartbeater.cc:347] Connected to a master server at 127.3.245.61:34469
I20260620 07:12:04.516979 11260 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.517647 11260 heartbeater.cc:511] Master 127.3.245.61:34469 requested a full tablet report, sending...
I20260620 07:12:04.517725 11258 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:04.517818 11258 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.517910 11259 heartbeater.cc:347] Connected to a master server at 127.3.245.60:34457
I20260620 07:12:04.517968 11259 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.518020 11258 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
I20260620 07:12:04.518102 11259 heartbeater.cc:511] Master 127.3.245.60:34457 requested a full tablet report, sending...
I20260620 07:12:04.518846 10752 ts_manager.cc:194] Registered new tserver with Master: 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:04.519140 10942 ts_manager.cc:194] Registered new tserver with Master: 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:04.519125 10158 ts_manager.cc:194] Re-registered known tserver with Master: 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:04.520002 10158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:59947
I20260620 07:12:04.533598 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.529437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41710)
0620 07:12:04.529561 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.529580 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.529723 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.530122 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.530126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.530153 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:04.530248 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.530254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.531391 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.531394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.532137 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.532145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.533216 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.533223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.533313 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.533335 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.533401 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.533477 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 07:12:04.533667 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.529629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.3.245.60:34457)
0620 07:12:04.529783 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.529789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.529807 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:04.529867 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.529872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.529950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.530069 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:04.530382 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.531251 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.532258 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.533072 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.533203 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.533398 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.533441 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.533517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:12:04.534492 10952 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 07:12:04.534945 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7563d4cea48b47cdab79bde300cbf61d. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.535115 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: LEARNER
W20260620 07:12:04.635638 11265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:04.635890 11265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:04.635931 11265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:04.639923 11265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:04.639978 11265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:04.640038 11265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:04.645145 11265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:42611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=46019
--tserver_master_addrs=127.3.245.62:34251,127.3.245.61:34469,127.3.245.60:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:04.646098 11265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:04.647079 11265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:04.656824 11273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.656814 11270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:04.656811 11271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:04.656919 11265 server_base.cc:1034] running on GCE node
I20260620 07:12:04.657574 11265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:04.658237 11265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:04.659431 11265 hybrid_clock.cc:648] HybridClock initialized: now 1781939524659414 us; error 33 us; skew 500 ppm
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:04.662613 11265 init.cc:377] Logged in from keytab as oryx/127.3.245.3@KRBTEST.COM (short username oryx)
I20260620 07:12:04.663542 11265 webserver.cc:460] Webserver started at http://127.3.245.3:46019/ using document root <none> and password file <none>
I20260620 07:12:04.664120 11265 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:04.664193 11265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:04.668020 11265 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:04.670425 11280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.671670 11265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:12:04.671773 11265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "970f2ebd3ae14132a5e63ccb53171632"
format_stamp: "Formatted at 2026-06-20 07:12:03 on dist-test-slave-ldfr"
server_key: "f8258e1bdbbcc414c13f8390b0fe29df"
server_key_iv: "ebd474269b892e8aa0e2b9dac56bc85a"
server_key_version: "encryptionkey@0"
I20260620 07:12:04.672281 11265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:04.684867 11265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:04.688133 11265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:04.688359 11265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:04.689033 11265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:04.689894 11265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:04.689939 11265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.690001 11265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:04.690033 11265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:04.698899 11265 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:42611
I20260620 07:12:04.698925 11393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:42611 every 8 connection(s)
I20260620 07:12:04.699993 11265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:04.704214 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11265
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.60@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.61@KRBTEST.COM
Jun 20 07:12:04 dist-test-slave-ldfr krb5kdc[10113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:04.718498 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.706277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.3.245.62:34251)
0620 07:12:04.706859 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.706863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.706877 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:04.706926 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.706929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.706985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.708113 (+ 1128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.708212 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709047 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.710252 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.711304 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.711413 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.713148 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.713154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.713159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.713205 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.715925 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.716402 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.716406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.716408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.716492 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.716840 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.716843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.716845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.717119 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.717276 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.718077 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.718271 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":97,"threads_started":1}
I20260620 07:12:04.720674 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.701940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:60403 (local address 127.3.245.60:34457)
0620 07:12:04.702065 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.702070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.703069 (+ 999us) server_negotiation.cc:407] Connection header received
0620 07:12:04.704186 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.704189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.704243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.704296 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.707646 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.708376 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709490 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709838 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709911 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.714808 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.714812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.714815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.714841 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.716765 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.719372 (+ 2607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.719375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.719377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.719458 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.719759 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.719762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.719764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.719974 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.720051 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.720368 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.720482 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":5504}
I20260620 07:12:04.720672 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.702069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:34501 (local address 127.3.245.61:34469)
0620 07:12:04.702175 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.702179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.703652 (+ 1473us) server_negotiation.cc:407] Connection header received
0620 07:12:04.704186 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.704189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.704243 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.704296 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.706379 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.707222 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709104 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709445 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709534 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.714387 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.714391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.714394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.714421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.717506 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.718136 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.718140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.718142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.718227 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.718675 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.718678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.718679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.718873 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.718963 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.720368 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.720482 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 07:12:04.721792 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.702348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.3.245.3:34501)
0620 07:12:04.703568 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.703578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.703944 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.704707 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.704717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.705441 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.706212 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.706234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.707387 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.707393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.708968 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.708977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709713 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.709919 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.710714 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.710749 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.714232 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.717657 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.717664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.717682 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.718029 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.718328 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.718333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.718337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.718540 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.718963 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.718971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.719336 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.719542 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.719907 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":318,"spinlock_wait_cycles":2944,"thread_start_us":115,"threads_started":1}
I20260620 07:12:04.722133 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.702348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.3.245.3:60403)
0620 07:12:04.702926 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.702976 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.703944 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.704707 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.704717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.705429 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.707475 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.707486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.708510 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.708513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.709363 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.709373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.710057 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.710061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.710168 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.710905 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.710920 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.714655 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.719031 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.719034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.719036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.719257 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.719563 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.719566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.719568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.719660 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.720105 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.720107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.720189 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.720871 (+ 682us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.721010 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":104,"spinlock_wait_cycles":2432,"thread_start_us":133,"threads_started":1}
I20260620 07:12:04.723253 11395 heartbeater.cc:347] Connected to a master server at 127.3.245.60:34457
I20260620 07:12:04.723476 11395 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.724026 11395 heartbeater.cc:511] Master 127.3.245.60:34457 requested a full tablet report, sending...
I20260620 07:12:04.724797 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.707717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.3.245.60:34457)
0620 07:12:04.713259 (+ 5542us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.713270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.713284 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:04.713456 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.713459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.713507 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.713582 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.714475 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.716434 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.717665 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.717994 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.718116 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.719812 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.719816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.719818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.719859 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.721883 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.722390 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.722396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.722401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.722486 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.722793 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.722796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.722798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.723005 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.723406 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.724407 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.724547 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5469,"spinlock_wait_cycles":4736,"thread_start_us":81,"threads_started":1}
I20260620 07:12:04.725173 11396 heartbeater.cc:347] Connected to a master server at 127.3.245.61:34469
I20260620 07:12:04.725214 10942 ts_manager.cc:194] Registered new tserver with Master: 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:04.725260 11396 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.725345 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.701940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:53519 (local address 127.3.245.62:34251)
0620 07:12:04.702202 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.702206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.703471 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 07:12:04.704190 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.704196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.704257 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.704321 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.706583 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.707365 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709340 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709647 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709750 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.716787 (+ 7037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.716794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.716800 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.716829 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.718488 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.722423 (+ 3935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.722427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.722428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.722504 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.723142 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.723146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.723149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.723389 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.723467 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.725053 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.725179 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260620 07:12:04.725423 11396 heartbeater.cc:511] Master 127.3.245.61:34469 requested a full tablet report, sending...
I20260620 07:12:04.726080 10752 ts_manager.cc:194] Registered new tserver with Master: 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:04.721796 10158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:04.728391 10158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:04.728843 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.702348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:53519)
0620 07:12:04.703370 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.703381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.703952 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.704710 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.704717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.705412 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:04.706446 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.706457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.707921 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.707924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.709230 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.709237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.709911 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.709914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.710020 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.710714 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:04.710736 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:04.716604 (+ 5868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:04.720238 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.720242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.720245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.722310 (+ 2065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.722613 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:04.722616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:04.722617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:04.722712 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:04.723523 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:04.723525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:04.724067 (+ 542us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.724978 (+ 911us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.725121 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":413,"spinlock_wait_cycles":57344,"thread_start_us":141,"threads_started":1}
I20260620 07:12:04.729151 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.707509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.3.245.61:34469)
0620 07:12:04.712332 (+ 4823us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.712336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.712349 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:12:04.712386 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.712391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.712506 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.712674 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:04.713655 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.714547 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.715810 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.716154 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.716258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.725599 (+ 9341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:04.725603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:04.725606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:04.725631 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:04.727311 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.727714 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.727718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.727720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.727810 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:04.728053 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:04.728056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:04.728058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:04.728263 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:04.728394 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.728696 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.728830 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4747,"thread_start_us":66,"threads_started":1}
I20260620 07:12:04.729761 11394 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34251
I20260620 07:12:04.729851 11394 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:04.729997 11394 heartbeater.cc:511] Master 127.3.245.62:34251 requested a full tablet report, sending...
I20260620 07:12:04.730623 10157 ts_manager.cc:194] Re-registered known tserver with Master: 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:04.731176 10157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:53519
I20260620 07:12:04.731997 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: NON_VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: true } } }
I20260620 07:12:04.732146 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.735615 11414 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: attempting to promote NON_VOTER 3ec43a8c6dec45a2afc7868958cd1a56 to VOTER
I20260620 07:12:04.735996 11414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:04.737213 10952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 LEARNER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 07:12:04.737587 11412 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 07:12:04.738847 10762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Refusing update from remote peer 7563d4cea48b47cdab79bde300cbf61d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 07:12:04.739250 11413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 07:12:04.739387 10718 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.739930 10952 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.740514 10762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.740988 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.741089 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.741065 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.741186 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.742166 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.742302 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:04.743834 11422 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.743957 11422 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:04.744181 10718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7563d4cea48b47cdab79bde300cbf61d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:04.744305 10718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:04.751132 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.744347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:39229 (local address 127.0.0.1:52662)
0620 07:12:04.744780 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.744797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.744887 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.746489 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.746494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.746514 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:04.746616 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.746624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.748058 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.748062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.749490 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.749500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750581 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.750585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.750751 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.750829 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.750893 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.750998 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":216}
I20260620 07:12:04.751403 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.746059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.3.245.3:42611)
0620 07:12:04.746360 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.746372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.746398 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:12:04.746471 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.746476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.746667 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.746981 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:04.747163 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.748188 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.749663 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750437 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.750562 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.751109 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.751164 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.751272 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260620 07:12:04.751410 11424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42611 (local address 127.0.0.1:42458)
0620 07:12:04.745875 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.745891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.745981 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.746961 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.746964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.746983 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:04.747067 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.747072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.748298 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.748301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.749506 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.749515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750565 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.750569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.750753 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.750831 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.750896 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.750990 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":57,"thread_start_us":154,"threads_started":1}
I20260620 07:12:04.751811 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.744745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.3.245.1:39229)
0620 07:12:04.745925 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.745940 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.745967 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:12:04.746049 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.746058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.746322 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.746498 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:04.746732 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.747939 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.749616 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750437 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.751273 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.751433 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.751549 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.751641 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":103,"threads_started":1}
I20260620 07:12:04.751880 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.745757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39773 (local address 127.0.0.1:55098)
0620 07:12:04.746747 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:04.746763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:04.746862 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:04.747560 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:04.747563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:04.747579 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:04.747672 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.747678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.749084 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.749089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.750009 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:04.750017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750910 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.750913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:04.751539 (+ 626us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.751562 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:12:04.751616 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 07:12:04.751719 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260620 07:12:04.752643 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:04.746363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.3.245.2:39773)
0620 07:12:04.747120 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:04.747132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:04.747153 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:12:04.747224 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:04.747229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:04.747411 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:04.748011 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:04.748150 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.748981 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.750132 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:04.750761 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:04.752145 (+ 1384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:04.752352 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:04.752398 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 07:12:04.752484 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":115,"threads_started":1}
I20260620 07:12:04.754804 11328 tablet_service.cc:1467] Processing CreateTablet for tablet 479e42ed6f6845588254f09647b78ac4 (DEFAULT_TABLE table=test-table [id=a0ccc092bd2d4af184d4db48e9fa7c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:04.754804 11055 tablet_service.cc:1467] Processing CreateTablet for tablet 479e42ed6f6845588254f09647b78ac4 (DEFAULT_TABLE table=test-table [id=a0ccc092bd2d4af184d4db48e9fa7c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:04.755918 11328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479e42ed6f6845588254f09647b78ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:04.755918 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479e42ed6f6845588254f09647b78ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:04.762616 11192 tablet_service.cc:1467] Processing CreateTablet for tablet 479e42ed6f6845588254f09647b78ac4 (DEFAULT_TABLE table=test-table [id=a0ccc092bd2d4af184d4db48e9fa7c55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:04.763623 11192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479e42ed6f6845588254f09647b78ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:04.764321 11428 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Bootstrap starting.
I20260620 07:12:04.765156 11429 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Bootstrap starting.
I20260620 07:12:04.767689 11429 tablet_bootstrap.cc:654] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:04.768671 11429 log.cc:824] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:04.770936 11429 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: No bootstrap required, opened a new log
I20260620 07:12:04.770987 11428 tablet_bootstrap.cc:654] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:04.771144 11429 ts_tablet_manager.cc:1397] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:12:04.771814 11428 log.cc:824] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:04.774269 11428 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: No bootstrap required, opened a new log
I20260620 07:12:04.774251 11432 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Bootstrap starting.
I20260620 07:12:04.774480 11428 ts_tablet_manager.cc:1397] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 07:12:04.775635 11429 raft_consensus.cc:348] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.775954 11429 raft_consensus.cc:374] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:04.776018 11429 raft_consensus.cc:729] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970f2ebd3ae14132a5e63ccb53171632, State: Initialized, Role: FOLLOWER
I20260620 07:12:04.776432 11429 consensus_queue.cc:260] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.776646 11432 tablet_bootstrap.cc:654] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:04.777357 11429 ts_tablet_manager.cc:1428] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260620 07:12:04.777684 11432 log.cc:824] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:04.777736 11394 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
I20260620 07:12:04.779780 11432 tablet_bootstrap.cc:492] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: No bootstrap required, opened a new log
I20260620 07:12:04.779992 11432 ts_tablet_manager.cc:1397] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:12:04.780779 11428 raft_consensus.cc:348] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.781047 11428 raft_consensus.cc:374] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:04.781116 11428 raft_consensus.cc:729] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf3bb64790a41ab96a65bd641703bdc, State: Initialized, Role: FOLLOWER
I20260620 07:12:04.781607 11428 consensus_queue.cc:260] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.782673 11428 ts_tablet_manager.cc:1428] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:12:04.783033 11121 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
I20260620 07:12:04.784780 11432 raft_consensus.cc:348] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.785054 11432 raft_consensus.cc:374] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:04.785113 11432 raft_consensus.cc:729] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66148eca92d744c4b67f998ff067f505, State: Initialized, Role: FOLLOWER
I20260620 07:12:04.785537 11432 consensus_queue.cc:260] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:04.786221 11258 heartbeater.cc:503] Master 127.3.245.62:34251 was elected leader, sending a full tablet report...
I20260620 07:12:04.786571 11432 ts_tablet_manager.cc:1428] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260620 07:12:04.795517 11124 tablet.cc:2369] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:12:04.952966 11397 tablet.cc:2369] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:12:04.997236 11261 tablet.cc:2369] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:05.053596 11435 raft_consensus.cc:482] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:05.053840 11435 raft_consensus.cc:504] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:05.055176 11435 leader_election.cc:288] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611), 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:05.059074 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.055563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:51457 (local address 127.3.245.3:42611)
0620 07:12:05.055698 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.055703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.055916 (+ 213us) server_negotiation.cc:407] Connection header received
0620 07:12:05.056082 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.056085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.056139 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.056218 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:05.056489 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.057060 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058110 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.058507 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058628 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.058689 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.058873 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.058937 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:12:05.059087 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.055631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48271 (local address 127.3.245.2:39773)
0620 07:12:05.055796 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.055803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.055845 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:12:05.056015 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.056021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.056069 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.056153 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:05.056392 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.056947 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058135 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.058519 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058671 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.058728 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.058894 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.058957 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 07:12:05.059139 11438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.055433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42611 (local address 127.3.245.1:51457)
0620 07:12:05.055814 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:05.055826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:05.055956 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:05.056226 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:05.056228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:05.056245 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:05.056380 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:05.056385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.057165 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.057172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:05.057983 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:05.057992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.058644 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:05.058743 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.058774 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:12:05.058858 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:12:05.058931 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":144,"thread_start_us":71,"threads_started":1}
I20260620 07:12:05.059156 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.055433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39773 (local address 127.3.245.1:48271)
0620 07:12:05.055770 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:05.055785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:05.055908 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:05.056164 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:05.056166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:05.056181 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:05.056291 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:05.056297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.057051 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.057056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:05.058003 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:05.058012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.058673 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.058677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:05.058779 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.058796 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:12:05.058880 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:12:05.058953 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":42,"thread_start_us":70,"threads_started":1}
I20260620 07:12:05.059713 11212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479e42ed6f6845588254f09647b78ac4" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66148eca92d744c4b67f998ff067f505" is_pre_election: true
I20260620 07:12:05.059999 11212 raft_consensus.cc:2454] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 0.
I20260620 07:12:05.060089 11348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479e42ed6f6845588254f09647b78ac4" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970f2ebd3ae14132a5e63ccb53171632" is_pre_election: true
I20260620 07:12:05.060379 11348 raft_consensus.cc:2454] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 0.
I20260620 07:12:05.060559 11011 leader_election.cc:302] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66148eca92d744c4b67f998ff067f505, bdf3bb64790a41ab96a65bd641703bdc; no voters:
I20260620 07:12:05.060853 11435 raft_consensus.cc:2781] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:05.060948 11435 raft_consensus.cc:482] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:05.061003 11435 raft_consensus.cc:3037] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:05.062119 11435 raft_consensus.cc:504] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:05.062493 11435 leader_election.cc:288] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 election: Requested vote from peers 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611), 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773)
I20260620 07:12:05.062788 11348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479e42ed6f6845588254f09647b78ac4" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970f2ebd3ae14132a5e63ccb53171632"
I20260620 07:12:05.062860 11212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479e42ed6f6845588254f09647b78ac4" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66148eca92d744c4b67f998ff067f505"
I20260620 07:12:05.062906 11348 raft_consensus.cc:3037] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:05.062976 11212 raft_consensus.cc:3037] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:05.064647 11348 raft_consensus.cc:2454] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 1.
I20260620 07:12:05.064647 11212 raft_consensus.cc:2454] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 1.
I20260620 07:12:05.064970 11011 leader_election.cc:302] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66148eca92d744c4b67f998ff067f505, bdf3bb64790a41ab96a65bd641703bdc; no voters:
I20260620 07:12:05.065271 11435 raft_consensus.cc:2781] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:05.065439 11435 raft_consensus.cc:686] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 1 LEADER]: Becoming Leader. State: Replica: bdf3bb64790a41ab96a65bd641703bdc, State: Running, Role: LEADER
I20260620 07:12:05.065789 11435 consensus_queue.cc:237] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } }
I20260620 07:12:05.069314 10158 catalog_manager.cc:5495] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1). New cstate: current_term: 1 leader_uuid: "bdf3bb64790a41ab96a65bd641703bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:05.114907 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.110842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.3.245.1:39229)
0620 07:12:05.110975 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.110979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.110992 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:12:05.111310 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.111314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.111383 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.111493 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:05.111791 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.112460 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.113411 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.113675 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.113777 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.114089 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:05.114533 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:05.114641 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.114711 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.114768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:12:05.120663 11348 raft_consensus.cc:1261] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Refusing update from remote peer bdf3bb64790a41ab96a65bd641703bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:05.120676 11212 raft_consensus.cc:1261] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Refusing update from remote peer bdf3bb64790a41ab96a65bd641703bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:05.121364 11439 consensus_queue.cc:1035] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:05.121532 11435 consensus_queue.cc:1035] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:05.131603 11443 mvcc.cc:205] Tried to move back new op lower bound from 7298824294887096320 to 7298824294671929344. Current Snapshot: MvccSnapshot[applied={T|T < 7298824294887096320}]
I20260620 07:12:05.142117 10157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:05.142282 10157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:05.146966 10157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7563d4cea48b47cdab79bde300cbf61d: Sending DeleteTablet for 3 replicas of tablet 479e42ed6f6845588254f09647b78ac4
I20260620 07:12:05.147846 11192 tablet_service.cc:1514] Processing DeleteTablet for tablet 479e42ed6f6845588254f09647b78ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:05 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:55098
I20260620 07:12:05.147928 11055 tablet_service.cc:1514] Processing DeleteTablet for tablet 479e42ed6f6845588254f09647b78ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:05 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:52662
I20260620 07:12:05.148319 11452 tablet_replica.cc:330] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: stopping tablet replica
I20260620 07:12:05.148355 11328 tablet_service.cc:1514] Processing DeleteTablet for tablet 479e42ed6f6845588254f09647b78ac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:05 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:42458
I20260620 07:12:05.148896 11454 tablet_replica.cc:330] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: stopping tablet replica
I20260620 07:12:05.149106 11452 raft_consensus.cc:2229] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:05.149183 11454 raft_consensus.cc:2229] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:05.149471 11454 raft_consensus.cc:2258] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:05.150106 11453 tablet_replica.cc:330] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: stopping tablet replica
I20260620 07:12:05.150804 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10126
I20260620 07:12:05.149435 11452 raft_consensus.cc:2258] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:05.151716 11454 ts_tablet_manager.cc:1905] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:05.153156 11452 ts_tablet_manager.cc:1905] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:05.153965 11454 ts_tablet_manager.cc:1918] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:05.154033 11454 log.cc:1196] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/479e42ed6f6845588254f09647b78ac4
I20260620 07:12:05.154258 11454 ts_tablet_manager.cc:1939] T 479e42ed6f6845588254f09647b78ac4 P 970f2ebd3ae14132a5e63ccb53171632: Deleting consensus metadata
I20260620 07:12:05.155647 11452 ts_tablet_manager.cc:1918] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:05.155745 11452 log.cc:1196] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/479e42ed6f6845588254f09647b78ac4
I20260620 07:12:05.156042 11452 ts_tablet_manager.cc:1939] T 479e42ed6f6845588254f09647b78ac4 P 66148eca92d744c4b67f998ff067f505: Deleting consensus metadata
W20260620 07:12:05.160681 11148 connection.cc:535] server connection from 127.0.0.1:55098 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 07:12:05.160693 11281 connection.cc:535] server connection from 127.0.0.1:42458 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:12:05.161247 11453 raft_consensus.cc:2229] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:05.161723 11453 raft_consensus.cc:2258] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:05.163208 11453 ts_tablet_manager.cc:1905] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:05.165920 11453 ts_tablet_manager.cc:1918] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:05.166009 11453 log.cc:1196] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/479e42ed6f6845588254f09647b78ac4
I20260620 07:12:05.166319 11453 ts_tablet_manager.cc:1939] T 479e42ed6f6845588254f09647b78ac4 P bdf3bb64790a41ab96a65bd641703bdc: Deleting consensus metadata
W20260620 07:12:05.167356 11010 connection.cc:441] server connection from 127.0.0.1:52662 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52662 (request call id 1) could send its response
I20260620 07:12:05.171921 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.150433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.3.245.61:34469)
0620 07:12:05.150929 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.150933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.150947 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:05.161126 (+ 10179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.161133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.161204 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.161265 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:05.162726 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.163538 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.164751 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.165093 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.165173 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.167677 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:05.167681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:05.167687 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:05.167720 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:05.169688 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.170230 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.170234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.170236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.170318 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.170601 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.170606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.170607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.170802 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:05.170895 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.171361 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.171742 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260620 07:12:05.177675 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.165283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.3.245.61:34469)
0620 07:12:05.165410 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.165414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.166717 (+ 1303us) server_negotiation.cc:407] Connection header received
0620 07:12:05.166920 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.166923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.166978 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.167035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:05.168540 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.169413 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.170658 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.170969 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.171033 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.172813 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:05.172817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:05.172828 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:05.172852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:05.174478 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.175407 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.175412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.175415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.175504 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.176074 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.176080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.176085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.176427 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:05.176583 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.177276 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.177456 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:12:05.182283 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.165978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.3.245.60:34457)
0620 07:12:05.166150 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.166156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.167571 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 07:12:05.167757 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.167760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.167806 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.167856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:05.168720 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.169511 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.170770 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.171112 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.171184 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.178616 (+ 7432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:05.178621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:05.178623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:05.178651 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:05.180458 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.180934 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.180939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.180941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.181022 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.181298 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.181301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.181303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.181564 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:05.181695 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.181916 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.182049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 07:12:05.185109 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:05.150204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.3.245.60:34457)
0620 07:12:05.150309 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:05.150314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:05.175399 (+ 25085us) server_negotiation.cc:407] Connection header received
0620 07:12:05.176233 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:05.176236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:05.176284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:05.176340 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:05.177779 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.178549 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.179553 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:05.179944 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:05.180044 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:05.181561 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:05.181564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:05.181567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:05.181589 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:05.183348 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.183810 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.183813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.183814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.183888 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:05.184158 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:05.184160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:05.184161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:05.184323 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:05.184418 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:05.184827 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 07:12:05.184975 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 07:12:06.134804 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.134090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:49465)
0620 07:12:06.134466 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.134571 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":102,"threads_started":1}
W20260620 07:12:06.135335 11121 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:34251 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.136061 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.134511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:43607)
0620 07:12:06.135839 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.135945 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":66,"threads_started":1}
I20260620 07:12:06.136482 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.135820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:44109)
0620 07:12:06.136330 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.136408 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":59,"threads_started":1}
W20260620 07:12:06.136519 11394 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:34251 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
W20260620 07:12:06.136822 11258 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:34251 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.647390 11466 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7563d4cea48b47cdab79bde300cbf61d)
I20260620 07:12:06.647388 11467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7563d4cea48b47cdab79bde300cbf61d)
I20260620 07:12:06.647578 11466 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:06.647610 11467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:06.648885 11466 leader_election.cc:288] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60:34457)
I20260620 07:12:06.649277 11467 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), be0a80ea8be443c2963696a562eda479 (127.3.245.61:34469)
I20260620 07:12:06.649598 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.648741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49652)
0620 07:12:06.649200 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.649518 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":62,"threads_started":1}
I20260620 07:12:06.649726 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.648711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49642)
0620 07:12:06.649164 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.649245 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":64,"threads_started":1}
W20260620 07:12:06.649798 10928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
W20260620 07:12:06.649900 10746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.650766 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.650519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49654)
0620 07:12:06.650642 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.650686 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 07:12:06.650902 10928 leader_election.cc:334] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.651000 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.650684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49664)
0620 07:12:06.650880 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.650931 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260620 07:12:06.651137 10746 leader_election.cc:334] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.653419 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.648788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:34469 (local address 127.0.0.1:39544)
0620 07:12:06.649261 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.649277 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:06.649377 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:06.649852 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:06.649855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:06.649911 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:06.650095 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:06.650105 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.651393 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.651399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:06.652281 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:06.652290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.652985 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.652988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:06.653092 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:06.653111 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:12:06.653210 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:12:06.653286 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":48,"threads_started":1}
I20260620 07:12:06.653409 11473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.649127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.3.245.61:34469)
0620 07:12:06.649502 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:06.649507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:06.649526 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:12:06.649591 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:06.649594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:06.649636 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:06.649720 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:06.650244 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.651290 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.652396 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.652869 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.652984 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:06.653046 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:06.653228 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 07:12:06.653296 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":71,"threads_started":1}
I20260620 07:12:06.653733 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.648751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:34457 (local address 127.0.0.1:41750)
0620 07:12:06.649507 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.649523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:06.649610 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:06.650303 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:06.650310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:06.650332 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:06.650438 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:06.650445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.651465 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.651479 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:06.652360 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:06.652371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.653225 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.653229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:06.653351 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:06.653371 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:12:06.653499 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 07:12:06.653583 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":1,"thread_start_us":44,"threads_started":1}
I20260620 07:12:06.654132 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.649092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.3.245.60:34457)
0620 07:12:06.649945 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:06.649950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:06.649967 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:06.650031 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:06.650034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:06.650082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:06.650182 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:06.650588 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.651344 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.652504 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:06.653096 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:06.653226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:06.653292 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:06.653657 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 07:12:06.653725 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":51,"threads_started":1}
I20260620 07:12:06.654240 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "be0a80ea8be443c2963696a562eda479" is_pre_election: true
I20260620 07:12:06.654492 10762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec43a8c6dec45a2afc7868958cd1a56 in term 1.
I20260620 07:12:06.654654 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be0a80ea8be443c2963696a562eda479" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" is_pre_election: true
I20260620 07:12:06.654983 10952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be0a80ea8be443c2963696a562eda479 in term 1.
I20260620 07:12:06.654938 10926 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ec43a8c6dec45a2afc7868958cd1a56, be0a80ea8be443c2963696a562eda479; no voters: 7563d4cea48b47cdab79bde300cbf61d
I20260620 07:12:06.655230 11467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 07:12:06.655285 11467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7563d4cea48b47cdab79bde300cbf61d)
I20260620 07:12:06.655329 11467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:12:06.655309 10747 leader_election.cc:302] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ec43a8c6dec45a2afc7868958cd1a56, be0a80ea8be443c2963696a562eda479; no voters: 7563d4cea48b47cdab79bde300cbf61d
I20260620 07:12:06.655534 11466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 07:12:06.655591 11466 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7563d4cea48b47cdab79bde300cbf61d)
I20260620 07:12:06.655629 11466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:12:06.656443 11467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:06.656667 11466 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:06.656956 11467 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 election: Requested vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), be0a80ea8be443c2963696a562eda479 (127.3.245.61:34469)
I20260620 07:12:06.657302 11466 leader_election.cc:288] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 election: Requested vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), 3ec43a8c6dec45a2afc7868958cd1a56 (127.3.245.60:34457)
I20260620 07:12:06.657380 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.657088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49668)
0620 07:12:06.657250 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.657305 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:12:06.657311 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "be0a80ea8be443c2963696a562eda479"
I20260620 07:12:06.657634 10762 raft_consensus.cc:2379] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ec43a8c6dec45a2afc7868958cd1a56 in current term 2: Already voted for candidate be0a80ea8be443c2963696a562eda479 in this term.
I20260620 07:12:06.657652 10952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "be0a80ea8be443c2963696a562eda479" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ec43a8c6dec45a2afc7868958cd1a56"
I20260620 07:12:06.657786 10952 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate be0a80ea8be443c2963696a562eda479 in current term 2: Already voted for candidate 3ec43a8c6dec45a2afc7868958cd1a56 in this term.
I20260620 07:12:06.657943 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.657141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49682)
0620 07:12:06.657245 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.657306 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 07:12:06.658524 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.657800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49686)
0620 07:12:06.658408 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.658460 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260620 07:12:06.658614 10928 leader_election.cc:334] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.658715 11469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:06.658455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49698)
0620 07:12:06.658590 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:06.658649 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:12:06.658718 10928 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ec43a8c6dec45a2afc7868958cd1a56; no voters: 7563d4cea48b47cdab79bde300cbf61d, be0a80ea8be443c2963696a562eda479
W20260620 07:12:06.658800 10746 leader_election.cc:334] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:06.658864 10746 leader_election.cc:302] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be0a80ea8be443c2963696a562eda479; no voters: 3ec43a8c6dec45a2afc7868958cd1a56, 7563d4cea48b47cdab79bde300cbf61d
I20260620 07:12:06.658993 11466 raft_consensus.cc:2726] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 07:12:06.659127 11467 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 07:12:07.136915 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:07.136153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:47327)
0620 07:12:07.136502 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:07.136577 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260620 07:12:07.138228 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:07.137289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:47233)
0620 07:12:07.138044 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:07.138112 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":179,"threads_started":1}
I20260620 07:12:07.138392 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:07.137592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:44347)
0620 07:12:07.138199 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:07.138261 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":79,"threads_started":1}
I20260620 07:12:08.138720 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.137750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:42551)
0620 07:12:08.138125 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.138204 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":88,"threads_started":1}
I20260620 07:12:08.139810 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.139310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:59517)
0620 07:12:08.139643 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.139707 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260620 07:12:08.139940 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.139312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:45163)
0620 07:12:08.139787 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.139847 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":78,"threads_started":1}
I20260620 07:12:08.313984 11482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:08.314129 11482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:08.314779 11482 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), be0a80ea8be443c2963696a562eda479 (127.3.245.61:34469)
I20260620 07:12:08.315353 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "be0a80ea8be443c2963696a562eda479" is_pre_election: true
I20260620 07:12:08.315554 10762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ec43a8c6dec45a2afc7868958cd1a56 in term 2.
I20260620 07:12:08.315656 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.315048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49972)
0620 07:12:08.315476 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.315549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":118,"threads_started":1}
I20260620 07:12:08.315951 10926 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ec43a8c6dec45a2afc7868958cd1a56, be0a80ea8be443c2963696a562eda479; no voters:
I20260620 07:12:08.316241 11482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 07:12:08.316310 11482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:08.316352 11482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:12:08.316656 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.316330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49980)
0620 07:12:08.316455 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.316533 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:12:08.316763 10928 leader_election.cc:334] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:08.317703 11482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:08.318054 11482 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 election: Requested vote from peers 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251), be0a80ea8be443c2963696a562eda479 (127.3.245.61:34469)
I20260620 07:12:08.318605 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.318372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:49994)
0620 07:12:08.318487 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.318538 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 07:12:08.318638 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "be0a80ea8be443c2963696a562eda479"
I20260620 07:12:08.318804 10762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:12:08.319938 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.319694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:50016)
0620 07:12:08.319815 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.319866 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:12:08.320029 10928 leader_election.cc:334] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
I20260620 07:12:08.320101 10762 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ec43a8c6dec45a2afc7868958cd1a56 in term 3.
I20260620 07:12:08.320413 10926 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ec43a8c6dec45a2afc7868958cd1a56, be0a80ea8be443c2963696a562eda479; no voters: 7563d4cea48b47cdab79bde300cbf61d
I20260620 07:12:08.320590 11482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:12:08.320829 11482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [term 3 LEADER]: Becoming Leader. State: Replica: 3ec43a8c6dec45a2afc7868958cd1a56, State: Running, Role: LEADER
I20260620 07:12:08.321142 11482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } }
I20260620 07:12:08.322438 11485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec43a8c6dec45a2afc7868958cd1a56. Latest consensus state: current_term: 3 leader_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:08.322569 11485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:08.322891 11487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:08.325341 11487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:08.325773 11487 catalog_manager.cc:1259] Loaded cluster ID: 5cb1b2eb4c974bd8a8f17109f2f80fc4
I20260620 07:12:08.325846 11487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:08.326431 11487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:08.326836 11487 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Loaded TSK: 0
I20260620 07:12:08.327544 11487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:08.405568 10942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:08.408718 10942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:08.411971 10762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [term 3 FOLLOWER]: Refusing update from remote peer 3ec43a8c6dec45a2afc7868958cd1a56: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 07:12:08.411999 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.411682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:50036)
0620 07:12:08.411814 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.411916 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:12:08.412585 11485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 07:12:08.412809 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.412531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.0.0.1:50052)
0620 07:12:08.412677 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.412732 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:12:08.412914 10928 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 -> Peer 7563d4cea48b47cdab79bde300cbf61d (127.3.245.62:34251): Couldn't send request to peer 7563d4cea48b47cdab79bde300cbf61d. Status: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 07:12:08.414115 11489 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec43a8c6dec45a2afc7868958cd1a56. Latest consensus state: current_term: 3 leader_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:08.414255 11489 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:08.415493 11485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:08.415669 11485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:08.415899 11489 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7563d4cea48b47cdab79bde300cbf61d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34251 } } peers { permanent_uuid: "be0a80ea8be443c2963696a562eda479" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 34469 } attrs { promote: false } } peers { permanent_uuid: "3ec43a8c6dec45a2afc7868958cd1a56" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 34457 } attrs { promote: false } } }
I20260620 07:12:08.416014 11489 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0a80ea8be443c2963696a562eda479 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:12:08.426847 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.422253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39773 (local address 127.0.0.1:55108)
0620 07:12:08.423049 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.423065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:08.423162 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:08.423514 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:08.423517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:08.423536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:08.423628 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.423633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.424657 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.424661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.425592 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.425601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.426399 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.426403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.426493 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.426518 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:12:08.426584 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:12:08.426667 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":5,"thread_start_us":87,"threads_started":1}
I20260620 07:12:08.427032 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.421596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:39229 (local address 127.0.0.1:52670)
0620 07:12:08.421735 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.421750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:08.421841 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:08.422299 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:08.422302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:08.422321 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:08.422433 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.422443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.423277 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.423280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.424513 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.424521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.426533 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.426536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.426658 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.426696 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:12:08.426784 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:12:08.426861 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":20}
I20260620 07:12:08.427351 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.421744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.3.245.1:39229)
0620 07:12:08.422030 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:08.422035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:08.422050 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:08.422106 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:08.422110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:08.422172 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:08.422262 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:08.422566 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.423110 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.425756 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.426382 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.426522 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.426587 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:08.426813 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 07:12:08.426893 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260620 07:12:08.427397 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.422364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.3.245.2:39773)
0620 07:12:08.422645 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:08.422650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:08.423158 (+ 508us) server_negotiation.cc:407] Connection header received
0620 07:12:08.423251 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:08.423256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:08.423309 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:08.423391 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:08.423758 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.424498 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.425695 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.426302 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.426406 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.426472 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:08.427034 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 07:12:08.427272 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260620 07:12:08.427927 11055 tablet_service.cc:1467] Processing CreateTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a (DEFAULT_TABLE table=test-table [id=3649c835ae8d46a9950624cef3e77ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:08.428052 11192 tablet_service.cc:1467] Processing CreateTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a (DEFAULT_TABLE table=test-table [id=3649c835ae8d46a9950624cef3e77ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:08.428256 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dba61d13ca34acbacf25db4a5a1151a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:08.428323 11192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dba61d13ca34acbacf25db4a5a1151a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:08.428406 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.422839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:42611 (local address 127.0.0.1:42474)
0620 07:12:08.423487 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.423499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:08.423586 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:08.424025 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:08.424028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:08.424043 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:08.424124 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.424136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.425048 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.425051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.426944 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:08.427064 (+ 120us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.428009 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.428013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:08.428107 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.428131 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:12:08.428214 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:12:08.428292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260620 07:12:08.428476 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.423165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.3.245.3:42611)
0620 07:12:08.423824 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:08.423827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:08.423838 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:12:08.423876 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:08.423878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:08.423914 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:08.424015 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:08.424238 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.424919 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.427182 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.427787 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.427889 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.427950 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:08.428264 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:12:08.428348 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":74,"threads_started":1}
I20260620 07:12:08.430703 11500 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: Bootstrap starting.
I20260620 07:12:08.430928 11328 tablet_service.cc:1467] Processing CreateTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a (DEFAULT_TABLE table=test-table [id=3649c835ae8d46a9950624cef3e77ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:08.430970 11501 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Bootstrap starting.
I20260620 07:12:08.431193 11328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dba61d13ca34acbacf25db4a5a1151a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:08.431639 11500 tablet_bootstrap.cc:654] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:08.431718 11501 tablet_bootstrap.cc:654] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:08.432926 11500 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: No bootstrap required, opened a new log
I20260620 07:12:08.432940 11501 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: No bootstrap required, opened a new log
I20260620 07:12:08.433019 11500 ts_tablet_manager.cc:1397] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:08.433485 11500 raft_consensus.cc:348] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.433575 11500 raft_consensus.cc:374] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:08.433595 11500 raft_consensus.cc:729] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf3bb64790a41ab96a65bd641703bdc, State: Initialized, Role: FOLLOWER
I20260620 07:12:08.433729 11500 consensus_queue.cc:260] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.434023 11504 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Bootstrap starting.
I20260620 07:12:08.434037 11501 ts_tablet_manager.cc:1397] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 07:12:08.434154 11500 ts_tablet_manager.cc:1428] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:08.434657 11501 raft_consensus.cc:348] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.434774 11501 raft_consensus.cc:374] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:08.434819 11501 raft_consensus.cc:729] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66148eca92d744c4b67f998ff067f505, State: Initialized, Role: FOLLOWER
I20260620 07:12:08.434924 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.434620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:55475)
0620 07:12:08.434781 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.434841 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:12:08.435022 11501 consensus_queue.cc:260] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.435238 11504 tablet_bootstrap.cc:654] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:08.435446 11501 ts_tablet_manager.cc:1428] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 07:12:08.436220 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.435899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:57373)
0620 07:12:08.436079 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.436141 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 07:12:08.436878 11504 tablet_bootstrap.cc:492] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: No bootstrap required, opened a new log
I20260620 07:12:08.437001 11504 ts_tablet_manager.cc:1397] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:08.437424 11504 raft_consensus.cc:348] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.437518 11504 raft_consensus.cc:374] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:08.437541 11504 raft_consensus.cc:729] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970f2ebd3ae14132a5e63ccb53171632, State: Initialized, Role: FOLLOWER
I20260620 07:12:08.437674 11504 consensus_queue.cc:260] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.437767 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.437472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:55597)
0620 07:12:08.437635 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.437691 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:12:08.438036 11122 heartbeater.cc:503] Master 127.3.245.60:34457 was elected leader, sending a full tablet report...
I20260620 07:12:08.438099 11504 ts_tablet_manager.cc:1428] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:08.438452 11259 heartbeater.cc:503] Master 127.3.245.60:34457 was elected leader, sending a full tablet report...
I20260620 07:12:08.439114 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.438823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:41893)
0620 07:12:08.438981 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.439040 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:12:08.439177 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.438923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:33961)
0620 07:12:08.439059 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.439100 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 07:12:08.441664 11395 heartbeater.cc:503] Master 127.3.245.60:34457 was elected leader, sending a full tablet report...
I20260620 07:12:08.441761 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.441460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:52149)
0620 07:12:08.441616 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.441681 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:08.517191 11505 raft_consensus.cc:482] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:08.517350 11505 raft_consensus.cc:504] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.517885 11505 leader_election.cc:288] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773), 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:08.518426 11212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dba61d13ca34acbacf25db4a5a1151a" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66148eca92d744c4b67f998ff067f505" is_pre_election: true
I20260620 07:12:08.518462 11348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dba61d13ca34acbacf25db4a5a1151a" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970f2ebd3ae14132a5e63ccb53171632" is_pre_election: true
I20260620 07:12:08.518594 11348 raft_consensus.cc:2454] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 0.
I20260620 07:12:08.518605 11212 raft_consensus.cc:2454] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 0.
I20260620 07:12:08.518980 11008 leader_election.cc:302] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66148eca92d744c4b67f998ff067f505, 970f2ebd3ae14132a5e63ccb53171632, bdf3bb64790a41ab96a65bd641703bdc; no voters:
I20260620 07:12:08.519227 11505 raft_consensus.cc:2781] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:08.519279 11505 raft_consensus.cc:482] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:08.519305 11505 raft_consensus.cc:3037] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:08.520897 11505 raft_consensus.cc:504] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.521363 11505 leader_election.cc:288] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 election: Requested vote from peers 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773), 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611)
I20260620 07:12:08.521732 11348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dba61d13ca34acbacf25db4a5a1151a" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970f2ebd3ae14132a5e63ccb53171632"
I20260620 07:12:08.521775 11212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dba61d13ca34acbacf25db4a5a1151a" candidate_uuid: "bdf3bb64790a41ab96a65bd641703bdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66148eca92d744c4b67f998ff067f505"
I20260620 07:12:08.521875 11348 raft_consensus.cc:3037] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:08.522001 11212 raft_consensus.cc:3037] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:08.522862 11348 raft_consensus.cc:2454] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 1.
I20260620 07:12:08.523110 11212 raft_consensus.cc:2454] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdf3bb64790a41ab96a65bd641703bdc in term 1.
I20260620 07:12:08.523169 11008 leader_election.cc:302] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 970f2ebd3ae14132a5e63ccb53171632, bdf3bb64790a41ab96a65bd641703bdc; no voters:
I20260620 07:12:08.523350 11505 raft_consensus.cc:2781] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:08.523610 11505 raft_consensus.cc:686] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 1 LEADER]: Becoming Leader. State: Replica: bdf3bb64790a41ab96a65bd641703bdc, State: Running, Role: LEADER
I20260620 07:12:08.523804 11505 consensus_queue.cc:237] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } }
I20260620 07:12:08.524459 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.524162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:44255)
0620 07:12:08.524278 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.524362 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 07:12:08.525704 10942 catalog_manager.cc:5495] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc reported cstate change: term changed from 0 to 1, leader changed from <none> to bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1). New cstate: current_term: 1 leader_uuid: "bdf3bb64790a41ab96a65bd641703bdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdf3bb64790a41ab96a65bd641703bdc" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 39229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:08.554351 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.551144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.3.245.1:39229)
0620 07:12:08.551315 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:08.551320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:08.551335 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:08.551410 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:08.551413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:08.551469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:08.551563 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:08.551827 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.552335 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.553290 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:08.553511 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:08.553632 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:08.553891 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:08.553965 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:08.554046 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:08.554141 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 07:12:08.554197 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 07:12:08.556516 11348 raft_consensus.cc:1261] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Refusing update from remote peer bdf3bb64790a41ab96a65bd641703bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:08.556546 11212 raft_consensus.cc:1261] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Refusing update from remote peer bdf3bb64790a41ab96a65bd641703bdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:08.557024 11509 consensus_queue.cc:1035] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "970f2ebd3ae14132a5e63ccb53171632" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:08.557219 11505 consensus_queue.cc:1035] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "66148eca92d744c4b67f998ff067f505" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:08.559044 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.558738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:50241)
0620 07:12:08.558896 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.558954 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 07:12:08.559213 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.558913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:43671)
0620 07:12:08.559079 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.559136 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 07:12:08.561667 11512 mvcc.cc:205] Tried to move back new op lower bound from 7298824308963225600 to 7298824308834938880. Current Snapshot: MvccSnapshot[applied={T|T < 7298824308963225600}]
I20260620 07:12:08.562539 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.562154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:60181)
0620 07:12:08.562323 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.562430 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 07:12:08.563453 11479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.563206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.1:37743)
0620 07:12:08.563338 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.563390 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:12:08.563764 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.563431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.2:53739)
0620 07:12:08.563626 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.563684 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 07:12:08.564435 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.564128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34251 (local address 127.3.245.3:56491)
0620 07:12:08.564312 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.564363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 07:12:08.570847 10941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:08.571010 10941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:08.575785 10941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ec43a8c6dec45a2afc7868958cd1a56: Sending DeleteTablet for 3 replicas of tablet 7dba61d13ca34acbacf25db4a5a1151a
I20260620 07:12:08.576639 11055 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:08 UTC) from {username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:52670
I20260620 07:12:08.576771 11192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:08 UTC) from {username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:55108
I20260620 07:12:08.577078 11520 tablet_replica.cc:330] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc: stopping tablet replica
I20260620 07:12:08.577135 11328 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dba61d13ca34acbacf25db4a5a1151a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:08 UTC) from {username='oryx', principal='oryx/127.3.245.60@KRBTEST.COM'} at 127.0.0.1:42474
I20260620 07:12:08.577181 11521 tablet_replica.cc:330] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: stopping tablet replica
I20260620 07:12:08.577271 11520 raft_consensus.cc:2229] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:08.577310 11521 raft_consensus.cc:2229] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:08.577437 11521 raft_consensus.cc:2258] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:08.577518 11520 raft_consensus.cc:2258] T 7dba61d13ca34acbacf25db4a5a1151a P bdf3bb64790a41ab96a65bd641703bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:08.577634 11522 tablet_replica.cc:330] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: stopping tablet replica
I20260620 07:12:08.577872 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10904
I20260620 07:12:08.578948 11522 raft_consensus.cc:2229] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:08.579094 11522 raft_consensus.cc:2258] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:08.579375 11521 ts_tablet_manager.cc:1905] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:08.579696 11522 ts_tablet_manager.cc:1905] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:08.581815 11522 ts_tablet_manager.cc:1918] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:08.581918 11522 log.cc:1196] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/7dba61d13ca34acbacf25db4a5a1151a
I20260620 07:12:08.582209 11522 ts_tablet_manager.cc:1939] T 7dba61d13ca34acbacf25db4a5a1151a P 970f2ebd3ae14132a5e63ccb53171632: Deleting consensus metadata
I20260620 07:12:08.582708 11521 ts_tablet_manager.cc:1918] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:08.582808 11521 log.cc:1196] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/7dba61d13ca34acbacf25db4a5a1151a
I20260620 07:12:08.582998 10926 catalog_manager.cc:4841] TS 970f2ebd3ae14132a5e63ccb53171632 (127.3.245.3:42611): tablet 7dba61d13ca34acbacf25db4a5a1151a (table test-table [id=3649c835ae8d46a9950624cef3e77ac5]) successfully deleted
I20260620 07:12:08.583104 11521 ts_tablet_manager.cc:1939] T 7dba61d13ca34acbacf25db4a5a1151a P 66148eca92d744c4b67f998ff067f505: Deleting consensus metadata
I20260620 07:12:08.583843 10929 catalog_manager.cc:4841] TS 66148eca92d744c4b67f998ff067f505 (127.3.245.2:39773): tablet 7dba61d13ca34acbacf25db4a5a1151a (table test-table [id=3649c835ae8d46a9950624cef3e77ac5]) successfully deleted
I20260620 07:12:08.587718 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11129
I20260620 07:12:08.587996 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:08.587663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:39229 (local address 127.0.0.1:52674)
0620 07:12:08.587826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:08.587891 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:39229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:12:08.588536 10928 catalog_manager.cc:4584] TS bdf3bb64790a41ab96a65bd641703bdc (127.3.245.1:39229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7dba61d13ca34acbacf25db4a5a1151a: Network error: Client connection negotiation failed: client connection to 127.3.245.1:39229: connect: Connection refused (error 111)
I20260620 07:12:08.593971 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11265
I20260620 07:12:08.599957 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10622
I20260620 07:12:08.606091 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 10726
2026-06-20T07:12:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7387 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:08 dist-test-slave-ldfr krb5kdc[11525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:08 dist-test-slave-ldfr krb5kdc[11525](info): set up 2 sockets
Jun 20 07:12:08 dist-test-slave-ldfr krb5kdc[11525](info): commencing operation
krb5kdc: starting...
W20260620 07:12:09.677546 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:09 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab.
Entry for principal oryx/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab.
I20260620 07:12:09.808734 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39897
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab
--principal=oryx/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:09.932431 11538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:09.932971 11538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:09.933084 11538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:09.937518 11538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:09.937628 11538 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 07:12:09.937657 11538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:09.937691 11538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:09.937727 11538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:09.937762 11538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:09.944191 11538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:39897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.62.keytab
--principal=oryx/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:09.945473 11538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:09.946673 11538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:09.956032 11543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:09.956019 11544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:09.956067 11538 server_base.cc:1034] running on GCE node
W20260620 07:12:09.956022 11546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:09.956923 11538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:09.958257 11538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:09.959503 11538 hybrid_clock.cc:648] HybridClock initialized: now 1781939529959474 us; error 68 us; skew 500 ppm
Jun 20 07:12:09 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939529, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:09.962733 11538 init.cc:377] Logged in from keytab as oryx/127.3.245.62@KRBTEST.COM (short username oryx)
I20260620 07:12:09.963750 11538 webserver.cc:460] Webserver started at http://127.3.245.62:44967/ using document root <none> and password file <none>
I20260620 07:12:09.964366 11538 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:09.964466 11538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:09.964778 11538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:09.967196 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "11644617fef44e2892b5841af6860378"
format_stamp: "Formatted at 2026-06-20 07:12:09 on dist-test-slave-ldfr"
server_key: "dd387648bed5b475d00d005b4aff3d7f"
server_key_iv: "529c452a316d577bbe7c5d364ec4481e"
server_key_version: "encryptionkey@0"
I20260620 07:12:09.967844 11538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "11644617fef44e2892b5841af6860378"
format_stamp: "Formatted at 2026-06-20 07:12:09 on dist-test-slave-ldfr"
server_key: "dd387648bed5b475d00d005b4aff3d7f"
server_key_iv: "529c452a316d577bbe7c5d364ec4481e"
server_key_version: "encryptionkey@0"
I20260620 07:12:09.972084 11538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:12:09.974838 11553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:09.976320 11538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:09.976486 11538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "11644617fef44e2892b5841af6860378"
format_stamp: "Formatted at 2026-06-20 07:12:09 on dist-test-slave-ldfr"
server_key: "dd387648bed5b475d00d005b4aff3d7f"
server_key_iv: "529c452a316d577bbe7c5d364ec4481e"
server_key_version: "encryptionkey@0"
I20260620 07:12:09.976689 11538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:10.007613 11538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:10.011276 11538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:10.011575 11538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:10.019896 11538 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:39897
I20260620 07:12:10.019963 11615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:39897 every 8 connection(s)
I20260620 07:12:10.021029 11538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:10.024159 11616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:10.027232 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11538
I20260620 07:12:10.027362 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:10.027679 4052 external_mini_cluster.cc:1428] Setting key f7125c6294ff9e5ffa272a7160d51755
I20260620 07:12:10.030731 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Bootstrap starting.
I20260620 07:12:10.033633 11616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:10.034611 11616 log.cc:824] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:10.037261 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: No bootstrap required, opened a new log
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:10.041668 11616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } }
I20260620 07:12:10.041988 11616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:10.042085 11616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11644617fef44e2892b5841af6860378, State: Initialized, Role: FOLLOWER
I20260620 07:12:10.042644 11616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } }
I20260620 07:12:10.042819 11616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:10.042922 11616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:10.043059 11616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:10.044031 11616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } }
I20260620 07:12:10.044175 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.029209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.3.245.62:39897)
0620 07:12:10.029703 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.029721 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.029759 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:12:10.030414 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.030432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.030813 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.031206 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.032133 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.033285 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.034511 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.035106 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.035295 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.038339 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.038344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.038359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.038390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.040671 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.041263 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.041269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.041274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.041401 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.041752 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.041754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.041755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.042303 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.042557 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.042891 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.043319 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
I20260620 07:12:10.044425 11616 leader_election.cc:302] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11644617fef44e2892b5841af6860378; no voters:
I20260620 07:12:10.044847 11616 leader_election.cc:288] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:10.044945 11621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:10.045152 11621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [term 1 LEADER]: Becoming Leader. State: Replica: 11644617fef44e2892b5841af6860378, State: Running, Role: LEADER
I20260620 07:12:10.045514 11621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } }
I20260620 07:12:10.046219 11616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:10.047534 11622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11644617fef44e2892b5841af6860378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } } }
I20260620 07:12:10.047572 11623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11644617fef44e2892b5841af6860378. Latest consensus state: current_term: 1 leader_uuid: "11644617fef44e2892b5841af6860378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11644617fef44e2892b5841af6860378" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 39897 } } }
I20260620 07:12:10.047740 11622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:10.047744 11623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:10.048172 11629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:10.051607 11629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:10.058175 11629 catalog_manager.cc:1347] Generated new cluster ID: b641c25f8d684286bc20a3d3254d3792
I20260620 07:12:10.058282 11629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:10.073200 11629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:10.074260 11629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:10.083406 11629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Generated new TSK 0
I20260620 07:12:10.084201 11629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
I20260620 07:12:10.125530 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39897
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:10.257530 11642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:10.257855 11642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:10.257915 11642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:10.262403 11642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:10.262481 11642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:10.262529 11642 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:12:10.262609 11642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:10.268464 11642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:10.269644 11642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:10.270666 11642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:10.281270 11650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:10.281213 11648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:10.281213 11647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:10.281577 11642 server_base.cc:1034] running on GCE node
I20260620 07:12:10.282191 11642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:10.282958 11642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:10.284163 11642 hybrid_clock.cc:648] HybridClock initialized: now 1781939530284119 us; error 56 us; skew 500 ppm
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:10.287542 11642 init.cc:377] Logged in from keytab as oryx/127.3.245.1@KRBTEST.COM (short username oryx)
I20260620 07:12:10.288582 11642 webserver.cc:460] Webserver started at http://127.3.245.1:42001/ using document root <none> and password file <none>
I20260620 07:12:10.289286 11642 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:10.289352 11642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:10.289636 11642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:10.291807 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "2957274c90674e78b0c35a7b03406b9d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "4287cb3525b16cd5b883d0c31698843c"
server_key_iv: "88dd55a6a5bc0fe6ecac08c14a94abdb"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.292398 11642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "2957274c90674e78b0c35a7b03406b9d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "4287cb3525b16cd5b883d0c31698843c"
server_key_iv: "88dd55a6a5bc0fe6ecac08c14a94abdb"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.296769 11642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:12:10.299902 11657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.301450 11642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:12:10.301647 11642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "2957274c90674e78b0c35a7b03406b9d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "4287cb3525b16cd5b883d0c31698843c"
server_key_iv: "88dd55a6a5bc0fe6ecac08c14a94abdb"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.301779 11642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:10.320591 11642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:10.324220 11642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:10.324442 11642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:10.326107 11642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:10.326181 11642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.326254 11642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:10.326294 11642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:10.338919 11642 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:44389
I20260620 07:12:10.338968 11772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:44389 every 8 connection(s)
I20260620 07:12:10.339902 11642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:10.341921 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.327060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:46529 (local address 127.3.245.62:39897)
0620 07:12:10.327194 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.327198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.328225 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 07:12:10.329463 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.329466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.329528 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.329651 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.331222 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.331946 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.333382 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.333842 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.333994 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.337696 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.337701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.337703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.337736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.339514 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.340155 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.340161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.340167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.340248 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.340697 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.340701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.340703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.340956 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.341078 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.341584 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.341739 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:12:10.342722 11666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.327568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.1:46529)
0620 07:12:10.328064 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.328110 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.329055 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.329842 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.329850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.330303 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.331049 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.331067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.332154 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.332157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.333234 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.333244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.334030 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.334043 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.334278 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.335091 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.335115 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.337457 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.339705 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.339711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.339726 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.340005 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.340348 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.340352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.340354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.340559 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.341068 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.341074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.341337 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.341525 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.341873 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
I20260620 07:12:10.344029 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11642
I20260620 07:12:10.344184 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:10.344287 11773 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39897
I20260620 07:12:10.344441 11773 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:10.344586 4052 external_mini_cluster.cc:1428] Setting key 68ade11f0f9b46ff92a9fae93cb2ae16
I20260620 07:12:10.345085 11773 heartbeater.cc:511] Master 127.3.245.62:39897 requested a full tablet report, sending...
I20260620 07:12:10.347009 11570 ts_manager.cc:194] Registered new tserver with Master: 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389)
I20260620 07:12:10.348460 11570 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:46529
I20260620 07:12:10.355710 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.346828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.1:36057)
0620 07:12:10.347195 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.347211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.347319 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.347619 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.347625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.347879 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.348039 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.348050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.349093 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.349099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.350134 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.350144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.350690 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.350693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.350769 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.351376 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.351388 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.351756 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.354095 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.354098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.354099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.354318 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.354652 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.354654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.354655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.354745 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.355217 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.355219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.355304 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.355391 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.355547 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260620 07:12:10.355829 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.346908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:36057 (local address 127.3.245.62:39897)
0620 07:12:10.347025 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.347029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.347274 (+ 245us) server_negotiation.cc:407] Connection header received
0620 07:12:10.347443 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.347446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.347488 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.347577 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.348214 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.348939 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.350259 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.350562 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.350671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.351921 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.351927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.351933 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.351965 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.353918 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.354428 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.354433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.354435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.354551 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.354840 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.354846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.354848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.355111 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.355218 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.355477 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.355628 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
I20260620 07:12:10.376272 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39897
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:10.494189 11782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:10.494477 11782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:10.494551 11782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:10.498790 11782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:10.498867 11782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:10.498910 11782 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:12:10.498986 11782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:10.504571 11782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:10.505867 11782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:10.507030 11782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:10.517524 11790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:10.517513 11787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:10.517737 11782 server_base.cc:1034] running on GCE node
W20260620 07:12:10.517515 11788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:10.518393 11782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:10.519114 11782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:10.520320 11782 hybrid_clock.cc:648] HybridClock initialized: now 1781939530520303 us; error 32 us; skew 500 ppm
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:10.523442 11782 init.cc:377] Logged in from keytab as oryx/127.3.245.2@KRBTEST.COM (short username oryx)
I20260620 07:12:10.524430 11782 webserver.cc:460] Webserver started at http://127.3.245.2:42233/ using document root <none> and password file <none>
I20260620 07:12:10.525102 11782 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:10.525177 11782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:10.525406 11782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:10.527554 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "e9a21cc577714a8fb414f2256bf0b20d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "f25bfa38b81ca47aeebcdb0df5627f5b"
server_key_iv: "685a99f67ba4b24d44d508e5e8ea4e0d"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.528128 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "e9a21cc577714a8fb414f2256bf0b20d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "f25bfa38b81ca47aeebcdb0df5627f5b"
server_key_iv: "685a99f67ba4b24d44d508e5e8ea4e0d"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.532217 11782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 07:12:10.535185 11797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.536609 11782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:10.536820 11782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "e9a21cc577714a8fb414f2256bf0b20d"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "f25bfa38b81ca47aeebcdb0df5627f5b"
server_key_iv: "685a99f67ba4b24d44d508e5e8ea4e0d"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.536959 11782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:10.554785 11782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:10.558210 11782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:10.558423 11782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:10.559844 11782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:10.559922 11782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.559995 11782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:10.560046 11782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:10.571053 11782 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:43533
I20260620 07:12:10.571727 11912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:43533 every 8 connection(s)
I20260620 07:12:10.572167 11782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:10.575711 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.560879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:45413 (local address 127.3.245.62:39897)
0620 07:12:10.561077 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.561080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.562157 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 07:12:10.563400 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.563406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.563477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.563584 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.565316 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.566198 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.568036 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.568338 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.568460 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.571659 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.571663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.571666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.571701 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.573480 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.574111 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.574115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.574117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.574206 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.574559 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.574564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.574566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.574768 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.574880 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.575401 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.575544 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 07:12:10.576584 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.561458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.2:45413)
0620 07:12:10.562006 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.562052 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.563135 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.563796 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.563808 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.564289 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.565117 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.565142 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.566322 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.566326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.567889 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.567904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.568503 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.568508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.568758 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.569534 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.569560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.571466 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.573621 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.573628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.573644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.573997 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.574303 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.574307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.574309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.574458 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.574878 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.574887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.575162 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.575360 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.575717 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260620 07:12:10.578411 11913 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39897
I20260620 07:12:10.578599 11913 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:10.579191 11913 heartbeater.cc:511] Master 127.3.245.62:39897 requested a full tablet report, sending...
I20260620 07:12:10.579878 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11782
I20260620 07:12:10.579998 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:10.580322 4052 external_mini_cluster.cc:1428] Setting key d871d01292368e50c496f127df485571
I20260620 07:12:10.580487 11570 ts_manager.cc:194] Registered new tserver with Master: e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533)
I20260620 07:12:10.581313 11570 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:45413
I20260620 07:12:10.588982 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.580974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:50557 (local address 127.3.245.62:39897)
0620 07:12:10.581160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.581165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.581370 (+ 205us) server_negotiation.cc:407] Connection header received
0620 07:12:10.581512 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.581517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.581567 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.581640 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.582307 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.583038 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.584111 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.584336 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.584448 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.585642 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.585645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.585647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.585672 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.587177 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.587640 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.587643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.587645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.587730 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.588030 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.588035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.588040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.588274 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.588384 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.588715 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.588833 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 07:12:10.588994 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.580898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.2:50557)
0620 07:12:10.581296 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.581313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.581421 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.581773 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.581776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.582035 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.582166 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.582174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.583199 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.583206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.583976 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.583985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.584454 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.584458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.584549 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.585142 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.585156 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.585493 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.587311 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.587314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.587317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.587529 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.587846 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.587849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.587851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.587941 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.588384 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.588388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.588490 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.588597 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.588792 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for oryx/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
I20260620 07:12:10.608208 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:39897
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:10.721872 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:10.722177 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:10.722222 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:10.726511 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:10.726584 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:10.726641 11922 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 07:12:10.726718 11922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:10.732209 11922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.245.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:10.733358 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:10.734449 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:10.744635 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:10.744712 11922 server_base.cc:1034] running on GCE node
W20260620 07:12:10.744717 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:10.744896 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:10.745540 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:10.746377 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:10.747577 11922 hybrid_clock.cc:648] HybridClock initialized: now 1781939530747553 us; error 39 us; skew 500 ppm
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:10.750639 11922 init.cc:377] Logged in from keytab as oryx/127.3.245.3@KRBTEST.COM (short username oryx)
I20260620 07:12:10.751655 11922 webserver.cc:460] Webserver started at http://127.3.245.3:42859/ using document root <none> and password file <none>
I20260620 07:12:10.752244 11922 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:10.752336 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:10.752576 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:10.754748 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "3e970aaf9b774a03b950c1be543d3db5"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "d03f2a5b369be69fd40932395964e66c"
server_key_iv: "a871d9562ae288de288afee4820e6981"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.755379 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "3e970aaf9b774a03b950c1be543d3db5"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "d03f2a5b369be69fd40932395964e66c"
server_key_iv: "a871d9562ae288de288afee4820e6981"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.759210 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 07:12:10.761797 11937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.763231 11922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:10.763396 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "3e970aaf9b774a03b950c1be543d3db5"
format_stamp: "Formatted at 2026-06-20 07:12:10 on dist-test-slave-ldfr"
server_key: "d03f2a5b369be69fd40932395964e66c"
server_key_iv: "a871d9562ae288de288afee4820e6981"
server_key_version: "encryptionkey@0"
I20260620 07:12:10.763530 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:10.796844 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:10.800305 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:10.800544 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:10.802096 11922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:10.802183 11922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:10.802258 11922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:10.802310 11922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:10.813982 11922 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:35753
I20260620 07:12:10.814091 12052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:35753 every 8 connection(s)
I20260620 07:12:10.815151 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:10.815367 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 11922
I20260620 07:12:10.815511 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:10.815848 4052 external_mini_cluster.cc:1428] Setting key fa1500711cb1ccb5fe231813734ecc46
I20260620 07:12:10.817592 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.803149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47435 (local address 127.3.245.62:39897)
0620 07:12:10.803280 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.803284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.804480 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 07:12:10.805990 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.805993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.806054 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.806140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.807925 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.808580 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.809963 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.810199 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.810319 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.813429 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.813432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.813435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.813462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.815138 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.815844 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.815848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.815849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.815939 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.816414 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.816416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.816418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.816659 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.816772 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.817315 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.817439 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:12:10.818392 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.803720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.3:47435)
0620 07:12:10.804321 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.804366 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.805761 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.806377 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.806386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.806945 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.807745 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.807767 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.808743 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.808746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.809848 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.809859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.810363 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.810368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.810558 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.811295 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.811313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.813206 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.815300 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.815307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.815324 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.815711 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.816050 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.816054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.816055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.816228 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.816780 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.816789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.817032 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.817266 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.817610 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":148,"threads_started":1}
I20260620 07:12:10.819530 12053 heartbeater.cc:347] Connected to a master server at 127.3.245.62:39897
I20260620 07:12:10.819679 12053 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:10.820169 12053 heartbeater.cc:511] Master 127.3.245.62:39897 requested a full tablet report, sending...
I20260620 07:12:10.821295 11570 ts_manager.cc:194] Registered new tserver with Master: 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:10.822007 11570 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:47435
I20260620 07:12:10.829128 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.821914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:39999 (local address 127.3.245.62:39897)
0620 07:12:10.822043 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.822049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.822294 (+ 245us) server_negotiation.cc:407] Connection header received
0620 07:12:10.822422 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.822425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.822464 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.822562 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.823062 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.823608 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.824437 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.824757 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.824863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.825900 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.825903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.825906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.825929 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.827235 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.827649 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.827654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.827656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.827738 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.828003 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.828008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.828010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.828248 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.828349 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.828813 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.828943 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:12:10.829139 12059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.821851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.3.245.3:39999)
0620 07:12:10.822198 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.822214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.822322 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.822574 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.822577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.822782 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:10.822930 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.822936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.823733 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.823736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.824320 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.824328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.824867 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.824870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.824961 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.825449 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:10.825459 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:10.825767 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:10.827343 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.827349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.827351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.827548 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.827825 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:10.827830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:10.827832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:10.827913 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:10.828401 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:10.828404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:10.828524 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.828679 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.828929 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":3,"thread_start_us":167,"threads_started":1}
I20260620 07:12:10.830009 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:12:10.854874 11619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:10.848384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.3.245.62:39897)
0620 07:12:10.848535 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.848538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.848549 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:12:10.848650 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.848659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.848727 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.848826 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.849606 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.850293 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.851201 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.851425 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.851511 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.854756 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35110: BlockingRecv error: recv got EOF from 127.0.0.1:35110 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 07:12:10 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939530, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:10.864735 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.856690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.3.245.62:39897)
0620 07:12:10.856841 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.856844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.856919 (+ 75us) server_negotiation.cc:407] Connection header received
0620 07:12:10.857045 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.857047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.857085 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.857149 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:10.857733 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.858278 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.859112 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.859335 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.859411 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.861746 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:10.861748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:10.861750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:10.861773 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:10.863202 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.863598 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.863601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.863602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.863677 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:10.863904 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:10.863907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:10.863908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:10.864088 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:10.864189 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.864370 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.864459 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:12:10.867750 11570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:10.870978 11570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:10.886642 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.880356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:35753 (local address 127.0.0.1:60194)
0620 07:12:10.881349 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.881365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.881446 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.882071 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.882077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.882102 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:10.882246 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.882253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.883539 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.883544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.884888 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.884899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.886016 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.886019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.886149 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886197 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.886308 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.886443 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260620 07:12:10.886785 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.881584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.3.245.1:44389)
0620 07:12:10.881824 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.881831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.881853 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:12:10.881926 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.881933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.882138 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.882602 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:10.882745 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.883874 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.885399 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.886178 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.886311 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886439 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.886538 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.886648 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260620 07:12:10.886965 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.881236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.3.245.2:43533)
0620 07:12:10.881672 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.881683 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.882307 (+ 624us) server_negotiation.cc:407] Connection header received
0620 07:12:10.882394 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.882399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.882594 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.882736 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:10.882956 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.884050 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.885431 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.886078 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.886176 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886325 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.886728 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.886838 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":104,"threads_started":1}
I20260620 07:12:10.886787 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.880355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:44389 (local address 127.0.0.1:35526)
0620 07:12:10.880802 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.880853 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.880985 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.882332 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.882334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.882352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:10.882455 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.882461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.884002 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.884005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.885266 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.885275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.886307 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.886310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.886395 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886415 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.886499 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.886576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":63,"thread_start_us":113,"threads_started":1}
I20260620 07:12:10.886936 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.881084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:43533 (local address 127.0.0.1:50172)
0620 07:12:10.882099 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:10.882112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:10.882197 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:10.882735 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:10.882738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:10.882757 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:10.882837 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.882842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.884153 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.884159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.885298 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:10.885308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.886380 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.886383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:10.886485 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886505 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:12:10.886572 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:12:10.886648 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":66,"thread_start_us":106,"threads_started":1}
I20260620 07:12:10.887603 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:10.881179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.3.245.3:35753)
0620 07:12:10.881549 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:10.881563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:10.881590 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:12:10.881669 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:10.881674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:10.881870 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:10.882419 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:10.882550 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.883428 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.885031 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:10.885872 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:10.885986 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:10.886157 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:10.887351 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 07:12:10.887468 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260620 07:12:10.890234 11987 tablet_service.cc:1467] Processing CreateTablet for tablet e263f415f46c4a2393f52edd23504547 (DEFAULT_TABLE table=test-table [id=eb00881c47214a0f84d451f65ad3bca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:10.890285 11847 tablet_service.cc:1467] Processing CreateTablet for tablet e263f415f46c4a2393f52edd23504547 (DEFAULT_TABLE table=test-table [id=eb00881c47214a0f84d451f65ad3bca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:10.890281 11707 tablet_service.cc:1467] Processing CreateTablet for tablet e263f415f46c4a2393f52edd23504547 (DEFAULT_TABLE table=test-table [id=eb00881c47214a0f84d451f65ad3bca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:10.891250 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263f415f46c4a2393f52edd23504547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:10.891285 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263f415f46c4a2393f52edd23504547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:10.891285 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e263f415f46c4a2393f52edd23504547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:10.898293 12081 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Bootstrap starting.
I20260620 07:12:10.899607 12083 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Bootstrap starting.
I20260620 07:12:10.899793 12082 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Bootstrap starting.
I20260620 07:12:10.900863 12081 tablet_bootstrap.cc:654] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:10.901655 12081 log.cc:824] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:10.901945 12082 tablet_bootstrap.cc:654] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:10.901945 12083 tablet_bootstrap.cc:654] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:10.902808 12082 log.cc:824] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:10.902808 12083 log.cc:824] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:10.903900 12081 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: No bootstrap required, opened a new log
I20260620 07:12:10.904119 12081 ts_tablet_manager.cc:1397] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:12:10.904718 12083 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: No bootstrap required, opened a new log
I20260620 07:12:10.904902 12082 tablet_bootstrap.cc:492] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: No bootstrap required, opened a new log
I20260620 07:12:10.904909 12083 ts_tablet_manager.cc:1397] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:12:10.905133 12082 ts_tablet_manager.cc:1397] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 07:12:10.907956 12083 raft_consensus.cc:348] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.908218 12083 raft_consensus.cc:374] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:10.908299 12083 raft_consensus.cc:729] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Initialized, Role: FOLLOWER
I20260620 07:12:10.908362 12081 raft_consensus.cc:348] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.908640 12081 raft_consensus.cc:374] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:10.908727 12081 raft_consensus.cc:729] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e970aaf9b774a03b950c1be543d3db5, State: Initialized, Role: FOLLOWER
I20260620 07:12:10.908833 12083 consensus_queue.cc:260] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.909163 12081 consensus_queue.cc:260] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.909583 11913 heartbeater.cc:503] Master 127.3.245.62:39897 was elected leader, sending a full tablet report...
I20260620 07:12:10.909967 12083 ts_tablet_manager.cc:1428] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:12:10.909668 12082 raft_consensus.cc:348] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.910137 12082 raft_consensus.cc:374] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:10.910192 12081 ts_tablet_manager.cc:1428] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:12:10.910193 12082 raft_consensus.cc:729] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Initialized, Role: FOLLOWER
I20260620 07:12:10.910507 12053 heartbeater.cc:503] Master 127.3.245.62:39897 was elected leader, sending a full tablet report...
I20260620 07:12:10.910593 12082 consensus_queue.cc:260] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:10.911566 12082 ts_tablet_manager.cc:1428] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:12:10.911975 11773 heartbeater.cc:503] Master 127.3.245.62:39897 was elected leader, sending a full tablet report...
W20260620 07:12:11.066609 12054 tablet.cc:2369] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:12:11.074363 11914 tablet.cc:2369] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:12:11.091531 11774 tablet.cc:2369] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:12:11.102358 12089 raft_consensus.cc:482] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:11.102613 12089 raft_consensus.cc:504] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:11.103864 12089 leader_election.cc:288] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753), e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533)
I20260620 07:12:11.108255 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.104200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:35753 (local address 127.3.245.1:60185)
0620 07:12:11.104588 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.104604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.104794 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.105138 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.105141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.105183 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:11.105329 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.105337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.106139 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.106145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.107140 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.107149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.107778 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.107781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.107881 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.107909 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.108007 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.108090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"spinlock_wait_cycles":24576,"thread_start_us":137,"threads_started":1}
I20260620 07:12:11.108228 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.104349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:60185 (local address 127.3.245.3:35753)
0620 07:12:11.104559 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.104565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.104902 (+ 337us) server_negotiation.cc:407] Connection header received
0620 07:12:11.104963 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.104966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.105023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.105123 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:11.105428 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.105986 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.107264 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.107656 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.107774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.107838 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.108023 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.108098 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:12:11.108533 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.104823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:37771 (local address 127.3.245.2:43533)
0620 07:12:11.105031 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.105036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.105137 (+ 101us) server_negotiation.cc:407] Connection header received
0620 07:12:11.105307 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.105313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.105364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.105461 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:11.105731 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.106451 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.107576 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.107970 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.108091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.108149 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.108337 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.108411 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:12:11.108546 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.104724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:43533 (local address 127.3.245.1:37771)
0620 07:12:11.105038 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.105051 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.105202 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.105480 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.105483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.105498 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:11.105610 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.105615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.106564 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.106569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.107446 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.107456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.108095 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.108098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.108208 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.108225 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.108309 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.108377 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":1408,"thread_start_us":116,"threads_started":1}
I20260620 07:12:11.109236 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e263f415f46c4a2393f52edd23504547" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a21cc577714a8fb414f2256bf0b20d" is_pre_election: true
I20260620 07:12:11.109244 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e263f415f46c4a2393f52edd23504547" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5" is_pre_election: true
I20260620 07:12:11.109565 11861 raft_consensus.cc:2454] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 0.
I20260620 07:12:11.109576 12007 raft_consensus.cc:2454] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 0.
I20260620 07:12:11.110075 11658 leader_election.cc:302] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.110357 12089 raft_consensus.cc:2781] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:11.110447 12089 raft_consensus.cc:482] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:11.110502 12089 raft_consensus.cc:3037] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.111538 12089 raft_consensus.cc:504] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:11.111882 12089 leader_election.cc:288] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 election: Requested vote from peers 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753), e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533)
I20260620 07:12:11.112213 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e263f415f46c4a2393f52edd23504547" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5"
I20260620 07:12:11.112294 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e263f415f46c4a2393f52edd23504547" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a21cc577714a8fb414f2256bf0b20d"
I20260620 07:12:11.112353 12007 raft_consensus.cc:3037] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.112434 11861 raft_consensus.cc:3037] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.113454 11861 raft_consensus.cc:2454] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 1.
I20260620 07:12:11.113816 11658 leader_election.cc:302] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.113960 12007 raft_consensus.cc:2454] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 1.
I20260620 07:12:11.114073 12089 raft_consensus.cc:2781] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:11.114315 12089 raft_consensus.cc:686] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 1 LEADER]: Becoming Leader. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Running, Role: LEADER
I20260620 07:12:11.114637 12089 consensus_queue.cc:237] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } }
I20260620 07:12:11.117926 11569 catalog_manager.cc:5495] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2957274c90674e78b0c35a7b03406b9d (127.3.245.1). New cstate: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:11.183322 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.179360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.3.245.1:44389)
0620 07:12:11.179704 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.179710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.179723 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:12:11.179803 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.179805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.179859 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.179976 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.180217 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.180801 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.181788 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.182012 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.182124 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.182419 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.182908 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.183041 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.183111 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.183169 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260620 07:12:11.189432 12007 raft_consensus.cc:1261] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Refusing update from remote peer 2957274c90674e78b0c35a7b03406b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.189432 11861 raft_consensus.cc:1261] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Refusing update from remote peer 2957274c90674e78b0c35a7b03406b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.190166 12092 consensus_queue.cc:1035] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.190399 12089 consensus_queue.cc:1035] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.198784 12097 mvcc.cc:205] Tried to move back new op lower bound from 7298824319744290816 to 7298824319448141824. Current Snapshot: MvccSnapshot[applied={T|T < 7298824319744290816}]
I20260620 07:12:11.208858 11570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.209045 11570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.211633 11570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Sending DeleteTablet for 3 replicas of tablet e263f415f46c4a2393f52edd23504547
I20260620 07:12:11.212528 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet e263f415f46c4a2393f52edd23504547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260620 07:12:11.212647 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet e263f415f46c4a2393f52edd23504547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60194
I20260620 07:12:11.212802 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet e263f415f46c4a2393f52edd23504547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:50172
I20260620 07:12:11.213181 12105 tablet_replica.cc:330] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: stopping tablet replica
I20260620 07:12:11.213533 12105 raft_consensus.cc:2229] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:11.213634 12107 tablet_replica.cc:330] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: stopping tablet replica
I20260620 07:12:11.213908 12107 raft_consensus.cc:2229] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.213964 12105 raft_consensus.cc:2258] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.214049 11570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 07:12:11.214296 12107 raft_consensus.cc:2258] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:12:11.214838 11570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:11.215541 12105 ts_tablet_manager.cc:1905] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.215593 12107 ts_tablet_manager.cc:1905] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.218225 12107 ts_tablet_manager.cc:1918] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.218255 12105 ts_tablet_manager.cc:1918] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.218343 12105 log.cc:1196] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/e263f415f46c4a2393f52edd23504547
I20260620 07:12:11.218420 12107 log.cc:1196] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/e263f415f46c4a2393f52edd23504547
I20260620 07:12:11.218689 12106 tablet_replica.cc:330] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: stopping tablet replica
I20260620 07:12:11.218784 12107 ts_tablet_manager.cc:1939] T e263f415f46c4a2393f52edd23504547 P e9a21cc577714a8fb414f2256bf0b20d: Deleting consensus metadata
I20260620 07:12:11.219053 12106 raft_consensus.cc:2229] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.219458 12106 raft_consensus.cc:2258] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.220189 12105 ts_tablet_manager.cc:1939] T e263f415f46c4a2393f52edd23504547 P 2957274c90674e78b0c35a7b03406b9d: Deleting consensus metadata
I20260620 07:12:11.220391 11707 tablet_service.cc:1467] Processing CreateTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 (DEFAULT_TABLE table=test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.220737 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c709eabe7794822bbe1aac44ac5de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.221177 12106 ts_tablet_manager.cc:1905] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.221240 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 (DEFAULT_TABLE table=test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.221467 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c709eabe7794822bbe1aac44ac5de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.223021 12082 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Bootstrap starting.
I20260620 07:12:11.223668 12106 ts_tablet_manager.cc:1918] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.223750 12106 log.cc:1196] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/e263f415f46c4a2393f52edd23504547
I20260620 07:12:11.224030 12082 tablet_bootstrap.cc:654] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.224052 12106 ts_tablet_manager.cc:1939] T e263f415f46c4a2393f52edd23504547 P 3e970aaf9b774a03b950c1be543d3db5: Deleting consensus metadata
I20260620 07:12:11.224134 11987 tablet_service.cc:1467] Processing CreateTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 (DEFAULT_TABLE table=test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.224429 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c709eabe7794822bbe1aac44ac5de39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.224546 11554 catalog_manager.cc:4841] TS e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533): tablet e263f415f46c4a2393f52edd23504547 (table test-table [id=eb00881c47214a0f84d451f65ad3bca8]) successfully deleted
I20260620 07:12:11.224690 12083 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Bootstrap starting.
I20260620 07:12:11.225581 12083 tablet_bootstrap.cc:654] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.226590 12081 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Bootstrap starting.
I20260620 07:12:11.226769 12083 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: No bootstrap required, opened a new log
I20260620 07:12:11.226838 12083 ts_tablet_manager.cc:1397] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:11.227049 12082 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: No bootstrap required, opened a new log
I20260620 07:12:11.227125 12082 ts_tablet_manager.cc:1397] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 07:12:11.227579 12081 tablet_bootstrap.cc:654] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.227557 12082 raft_consensus.cc:348] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.227594 12083 raft_consensus.cc:348] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.227681 12082 raft_consensus.cc:374] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.227694 12083 raft_consensus.cc:374] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.227715 12082 raft_consensus.cc:729] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.227732 12083 raft_consensus.cc:729] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.227833 12082 consensus_queue.cc:260] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.227859 12083 consensus_queue.cc:260] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.228122 12082 ts_tablet_manager.cc:1428] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.228175 11554 catalog_manager.cc:4841] TS 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753): tablet e263f415f46c4a2393f52edd23504547 (table test-table [id=eb00881c47214a0f84d451f65ad3bca8]) successfully deleted
I20260620 07:12:11.229044 12083 ts_tablet_manager.cc:1428] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:11.230149 12081 tablet_bootstrap.cc:492] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: No bootstrap required, opened a new log
I20260620 07:12:11.230240 12081 ts_tablet_manager.cc:1397] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:12:11.230676 12081 raft_consensus.cc:348] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.230792 12081 raft_consensus.cc:374] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.230831 12081 raft_consensus.cc:729] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e970aaf9b774a03b950c1be543d3db5, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.230973 12081 consensus_queue.cc:260] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.231232 12081 ts_tablet_manager.cc:1428] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:12:11.232309 11557 catalog_manager.cc:4841] TS 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389): tablet e263f415f46c4a2393f52edd23504547 (table test-table [id=eb00881c47214a0f84d451f65ad3bca8]) successfully deleted
I20260620 07:12:11.293936 12087 raft_consensus.cc:482] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:11.294112 12087 raft_consensus.cc:504] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.295462 12087 leader_election.cc:288] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.300140 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.295794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:49839 (local address 127.3.245.1:44389)
0620 07:12:11.295932 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.295937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.296057 (+ 120us) server_negotiation.cc:407] Connection header received
0620 07:12:11.296271 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.296277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.296363 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.296415 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:11.296903 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.297714 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.298954 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.299493 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.299597 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.299665 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.299932 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.300012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:12:11.300180 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.295631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:44389 (local address 127.3.245.2:49839)
0620 07:12:11.295963 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.295978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.296137 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.296569 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.296571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.296589 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:11.296783 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.296789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.297870 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.297873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.298823 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.298833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.299650 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.299654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.299772 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.299802 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.299893 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.299971 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":76,"thread_start_us":90,"threads_started":1}
I20260620 07:12:11.300866 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c709eabe7794822bbe1aac44ac5de39" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d" is_pre_election: true
I20260620 07:12:11.301120 11727 raft_consensus.cc:2454] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:11.301558 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.296108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:43515 (local address 127.3.245.3:35753)
0620 07:12:11.296226 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.296233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.296248 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:11.296388 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.296395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.296451 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.296526 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:11.297003 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.297713 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.299273 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.299815 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.299949 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.300005 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.301348 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.301445 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:12:11.301589 11801 leader_election.cc:302] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.301863 12087 raft_consensus.cc:2781] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:11.301951 12087 raft_consensus.cc:482] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:11.301990 12087 raft_consensus.cc:3037] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.302033 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.295733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:35753 (local address 127.3.245.2:43515)
0620 07:12:11.296154 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.296169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.296269 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.296559 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.296562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.296580 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:11.296825 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.296832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.297833 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.297837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.299129 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.299141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.300413 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.300416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.300539 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.300562 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.301748 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.301829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":96,"thread_start_us":113,"threads_started":1}
I20260620 07:12:11.302495 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c709eabe7794822bbe1aac44ac5de39" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5" is_pre_election: true
I20260620 07:12:11.302630 12007 raft_consensus.cc:2454] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:11.303159 12087 raft_consensus.cc:504] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.303543 12087 leader_election.cc:288] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Requested vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.303803 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c709eabe7794822bbe1aac44ac5de39" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d"
I20260620 07:12:11.303926 11727 raft_consensus.cc:3037] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.303915 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c709eabe7794822bbe1aac44ac5de39" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5"
I20260620 07:12:11.304054 12007 raft_consensus.cc:3037] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.304946 11727 raft_consensus.cc:2454] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:11.305249 11801 leader_election.cc:302] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.305297 12007 raft_consensus.cc:2454] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:11.305428 12087 raft_consensus.cc:2781] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:11.305635 12087 raft_consensus.cc:686] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Becoming Leader. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Running, Role: LEADER
I20260620 07:12:11.305868 12087 consensus_queue.cc:237] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.308379 11569 catalog_manager.cc:5495] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2). New cstate: current_term: 1 leader_uuid: "e9a21cc577714a8fb414f2256bf0b20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } health_report { overall_health: UNKNOWN } } }
Jun 20 07:12:11 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939529, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.62@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:11.363903 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.354489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:39897 (local address 127.0.0.1:35132)
0620 07:12:11.354777 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.354793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.354941 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.355212 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.355215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.355689 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:11.355779 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.355786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.356795 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.356801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.357350 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.357359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.357810 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.357813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.357896 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.358474 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:11.358486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:11.360222 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:11.362289 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:11.362297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:11.362303 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:11.362555 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:11.362885 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:11.362887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:11.362888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:11.362970 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:11.363409 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:11.363415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:11.363526 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.363636 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.363761 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260620 07:12:11.364041 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.354505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.3.245.62:39897)
0620 07:12:11.354634 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.354641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.354880 (+ 239us) server_negotiation.cc:407] Connection header received
0620 07:12:11.355070 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.355072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.355110 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.355196 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:11.355895 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.356685 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.357465 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.357698 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.357803 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.360410 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:11.360415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:11.360418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:11.360448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:11.362164 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:11.362678 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:11.362682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:11.362684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:11.362779 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:11.363052 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:11.363055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:11.363057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:11.363296 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:11.363403 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.363713 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.363840 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:12:11.368428 11569 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35132:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 07:12:11.369257 11569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:11.374687 11707 tablet_service.cc:1467] Processing CreateTablet for tablet d45c4e605ab74d60a08141c3d127eec4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06a5ee08a8b24f978fdfd18b75a5afc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:12:11.375128 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c4e605ab74d60a08141c3d127eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.375002 11847 tablet_service.cc:1467] Processing CreateTablet for tablet d45c4e605ab74d60a08141c3d127eec4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06a5ee08a8b24f978fdfd18b75a5afc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:12:11.375350 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c4e605ab74d60a08141c3d127eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.378130 12082 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: Bootstrap starting.
I20260620 07:12:11.378379 11987 tablet_service.cc:1467] Processing CreateTablet for tablet d45c4e605ab74d60a08141c3d127eec4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06a5ee08a8b24f978fdfd18b75a5afc2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 07:12:11.378670 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45c4e605ab74d60a08141c3d127eec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.379092 12082 tablet_bootstrap.cc:654] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.380499 12082 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: No bootstrap required, opened a new log
I20260620 07:12:11.380596 12082 ts_tablet_manager.cc:1397] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 07:12:11.381072 12082 raft_consensus.cc:348] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.381220 12082 raft_consensus.cc:374] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.381254 12082 raft_consensus.cc:729] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.381319 12081 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: Bootstrap starting.
I20260620 07:12:11.381410 12082 consensus_queue.cc:260] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.381696 12082 ts_tablet_manager.cc:1428] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.382323 12081 tablet_bootstrap.cc:654] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.382321 12083 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: Bootstrap starting.
I20260620 07:12:11.382526 12101 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.382668 12101 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.383345 12083 tablet_bootstrap.cc:654] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.384657 12083 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: No bootstrap required, opened a new log
I20260620 07:12:11.384744 12083 ts_tablet_manager.cc:1397] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 07:12:11.384750 12081 tablet_bootstrap.cc:492] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: No bootstrap required, opened a new log
I20260620 07:12:11.384816 12081 ts_tablet_manager.cc:1397] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:12:11.385211 12083 raft_consensus.cc:348] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.385214 12081 raft_consensus.cc:348] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.385324 12083 raft_consensus.cc:374] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.385324 12081 raft_consensus.cc:374] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.385360 12081 raft_consensus.cc:729] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e970aaf9b774a03b950c1be543d3db5, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.385360 12083 raft_consensus.cc:729] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.385489 12083 consensus_queue.cc:260] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.385480 12081 consensus_queue.cc:260] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.385747 12083 ts_tablet_manager.cc:1428] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.385747 12081 ts_tablet_manager.cc:1428] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.385962 12087 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.386106 12087 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.386948 12088 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.387100 12088 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.518103 12101 raft_consensus.cc:482] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:11.518244 12101 raft_consensus.cc:504] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.518738 12101 leader_election.cc:288] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.519436 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c4e605ab74d60a08141c3d127eec4" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5" is_pre_election: true
I20260620 07:12:11.519433 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c4e605ab74d60a08141c3d127eec4" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a21cc577714a8fb414f2256bf0b20d" is_pre_election: true
I20260620 07:12:11.519634 12007 raft_consensus.cc:2454] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 0.
I20260620 07:12:11.519650 11861 raft_consensus.cc:2454] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 0.
I20260620 07:12:11.519940 11658 leader_election.cc:302] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, 3e970aaf9b774a03b950c1be543d3db5; no voters:
I20260620 07:12:11.520124 12101 raft_consensus.cc:2781] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:11.520196 12101 raft_consensus.cc:482] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:11.520247 12101 raft_consensus.cc:3037] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.521358 12101 raft_consensus.cc:504] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.521695 12101 leader_election.cc:288] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 election: Requested vote from peers e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.522133 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c4e605ab74d60a08141c3d127eec4" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5"
I20260620 07:12:11.522133 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45c4e605ab74d60a08141c3d127eec4" candidate_uuid: "2957274c90674e78b0c35a7b03406b9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a21cc577714a8fb414f2256bf0b20d"
I20260620 07:12:11.522270 12007 raft_consensus.cc:3037] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.522269 11861 raft_consensus.cc:3037] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.523375 12007 raft_consensus.cc:2454] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 1.
I20260620 07:12:11.523375 11861 raft_consensus.cc:2454] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2957274c90674e78b0c35a7b03406b9d in term 1.
I20260620 07:12:11.523699 11658 leader_election.cc:302] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, 3e970aaf9b774a03b950c1be543d3db5; no voters:
I20260620 07:12:11.523981 12101 raft_consensus.cc:2781] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:11.524144 12101 raft_consensus.cc:686] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [term 1 LEADER]: Becoming Leader. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Running, Role: LEADER
I20260620 07:12:11.524288 12101 consensus_queue.cc:237] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.525058 12089 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: TxnStatusTablet state changed. Reason: New leader 2957274c90674e78b0c35a7b03406b9d. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.525177 12089 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:12:11.525442 12125 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 07:12:11.525614 12125 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 07:12:11.526197 11570 catalog_manager.cc:5495] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2957274c90674e78b0c35a7b03406b9d (127.3.245.1). New cstate: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:11.546478 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.542463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.3.245.1:44389)
0620 07:12:11.542590 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.542595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.542612 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:11.542745 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.542748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.542803 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.542892 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.543136 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.543916 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.545039 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.545531 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.545665 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.545991 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.546088 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.546172 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.546278 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.546333 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:12:11.546478 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.542362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:44389 (local address 127.0.0.1:35534)
0620 07:12:11.542499 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.542546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.542632 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.542932 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.542935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.542941 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:12:11.543035 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.543040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.544055 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.544058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.544878 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.544893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.545666 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.545669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.545788 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.545835 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.546177 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.546194 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.546254 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.546352 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 07:12:11.549530 11861 raft_consensus.cc:1261] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Refusing update from remote peer 2957274c90674e78b0c35a7b03406b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.549546 12007 raft_consensus.cc:1261] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Refusing update from remote peer 2957274c90674e78b0c35a7b03406b9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.549996 12089 consensus_queue.cc:1035] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.550367 12089 consensus_queue.cc:1035] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.551662 12087 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: TxnStatusTablet state changed. Reason: New leader 2957274c90674e78b0c35a7b03406b9d. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.551782 12087 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.551939 12088 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: TxnStatusTablet state changed. Reason: New leader 2957274c90674e78b0c35a7b03406b9d. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.552052 12088 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.553289 12089 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.553385 12089 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:12:11.553459 12095 mvcc.cc:205] Tried to move back new op lower bound from 7298824321221836800 to 7298824321125244928. Current Snapshot: MvccSnapshot[applied={T|T < 7298824321221836800}]
I20260620 07:12:11.555094 12088 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.555207 12088 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 3e970aaf9b774a03b950c1be543d3db5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.558745 12092 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.558861 12092 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P 2957274c90674e78b0c35a7b03406b9d: This TxnStatusTablet replica's current role is: LEADER
I20260620 07:12:11.559546 12087 tablet_replica.cc:439] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2957274c90674e78b0c35a7b03406b9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } } }
I20260620 07:12:11.559669 12087 tablet_replica.cc:442] T d45c4e605ab74d60a08141c3d127eec4 P e9a21cc577714a8fb414f2256bf0b20d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 07:12:11.564185 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.559024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.3.245.2:43533)
0620 07:12:11.559128 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.559133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.559145 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:12:11.559222 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.559225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.559285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.559390 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.560992 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.561769 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.562604 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.562936 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.563047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.563261 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.563829 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.563939 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.563993 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.564049 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:12:11.575142 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.571801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:59979 (local address 127.3.245.1:44389)
0620 07:12:11.572055 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.572060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.572101 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:12:11.572268 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.572271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.572330 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.572411 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.572693 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.573200 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.574082 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.574303 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.574411 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.574700 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.574777 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.574856 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.574941 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.574997 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260620 07:12:11.575156 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.571724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:44389 (local address 127.3.245.2:59979)
0620 07:12:11.572030 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.572045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.572160 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.572421 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.572424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.572432 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:12:11.572560 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.572565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.573309 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.573314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.573960 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.573968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.574413 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.574416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.574522 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.574563 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.574859 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.574867 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.574932 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.575005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260620 07:12:11.583401 12007 raft_consensus.cc:1261] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.583617 11727 raft_consensus.cc:1261] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.583868 12087 consensus_queue.cc:1035] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.584060 12113 consensus_queue.cc:1035] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.587235 12096 mvcc.cc:205] Tried to move back new op lower bound from 7298824321361219584 to 7298824320231563264. Current Snapshot: MvccSnapshot[applied={T|T < 7298824321361219584}]
I20260620 07:12:11.607995 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.604136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:35861 (local address 127.3.245.2:43533)
0620 07:12:11.604310 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.604315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.604394 (+ 79us) server_negotiation.cc:407] Connection header received
0620 07:12:11.604509 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.604512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.604562 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.604667 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.604975 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.605652 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.606880 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.607152 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.607262 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.607540 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.607620 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.607695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.607781 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.607829 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 07:12:11.607995 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.604018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:43533 (local address 127.3.245.1:35861)
0620 07:12:11.604326 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:11.604339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:11.604434 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:11.604681 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:11.604684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:11.604692 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 07:12:11.604816 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.604826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.605975 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.605977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.606761 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:11.606771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.607260 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.607264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:11.607348 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.607403 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.607698 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.607707 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:12:11.607771 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 07:12:11.607852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
I20260620 07:12:11.637742 11570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.637934 11570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.640385 11570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Sending DeleteTablet for 3 replicas of tablet 5c709eabe7794822bbe1aac44ac5de39
I20260620 07:12:11.641014 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260620 07:12:11.641213 12105 tablet_replica.cc:330] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: stopping tablet replica
I20260620 07:12:11.641278 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60194
I20260620 07:12:11.641347 12105 raft_consensus.cc:2229] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.641489 12106 tablet_replica.cc:330] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: stopping tablet replica
I20260620 07:12:11.641489 12105 raft_consensus.cc:2258] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.641599 12106 raft_consensus.cc:2229] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.641649 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c709eabe7794822bbe1aac44ac5de39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:50172
I20260620 07:12:11.641705 12106 raft_consensus.cc:2258] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.641826 12107 tablet_replica.cc:330] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: stopping tablet replica
I20260620 07:12:11.641940 12107 raft_consensus.cc:2229] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:11.642261 12107 raft_consensus.cc:2258] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.642736 12105 ts_tablet_manager.cc:1905] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.643134 12106 ts_tablet_manager.cc:1905] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.645784 12106 ts_tablet_manager.cc:1918] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:12:11.645805 12105 ts_tablet_manager.cc:1918] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:12:11.645869 12105 log.cc:1196] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/5c709eabe7794822bbe1aac44ac5de39
I20260620 07:12:11.645947 12106 log.cc:1196] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/5c709eabe7794822bbe1aac44ac5de39
I20260620 07:12:11.646153 12105 ts_tablet_manager.cc:1939] T 5c709eabe7794822bbe1aac44ac5de39 P 2957274c90674e78b0c35a7b03406b9d: Deleting consensus metadata
I20260620 07:12:11.646188 12106 ts_tablet_manager.cc:1939] T 5c709eabe7794822bbe1aac44ac5de39 P 3e970aaf9b774a03b950c1be543d3db5: Deleting consensus metadata
I20260620 07:12:11.646790 12107 ts_tablet_manager.cc:1905] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.646950 11554 catalog_manager.cc:4841] TS 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753): tablet 5c709eabe7794822bbe1aac44ac5de39 (table test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]) successfully deleted
I20260620 07:12:11.647253 11557 catalog_manager.cc:4841] TS 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389): tablet 5c709eabe7794822bbe1aac44ac5de39 (table test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]) successfully deleted
I20260620 07:12:11.649585 12107 ts_tablet_manager.cc:1918] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:12:11.649653 12107 log.cc:1196] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/5c709eabe7794822bbe1aac44ac5de39
I20260620 07:12:11.649896 12107 ts_tablet_manager.cc:1939] T 5c709eabe7794822bbe1aac44ac5de39 P e9a21cc577714a8fb414f2256bf0b20d: Deleting consensus metadata
I20260620 07:12:11.650616 11554 catalog_manager.cc:4841] TS e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533): tablet 5c709eabe7794822bbe1aac44ac5de39 (table test-table [id=8a9e28274fc04e358d5dd28ebf5cb718]) successfully deleted
Jun 20 07:12:11 dist-test-slave-ldfr krb5kdc[11525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 07:12:11 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:11 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:12:11.685272 11619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:11.677152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.3.245.62:39897)
0620 07:12:11.677277 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.677282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.677574 (+ 292us) server_negotiation.cc:407] Connection header received
0620 07:12:11.677687 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.677691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.677752 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.677849 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:11.678771 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.679694 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.680984 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.681317 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.681425 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.685139 (+ 3714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35146: BlockingRecv error: recv got EOF from 127.0.0.1:35146 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 07:12:11 dist-test-slave-ldfr krb5kdc[11525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM
I20260620 07:12:11.697994 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.687701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.3.245.62:39897)
0620 07:12:11.687804 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.687808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.687943 (+ 135us) server_negotiation.cc:407] Connection header received
0620 07:12:11.688059 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.688062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.688109 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.688195 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:11.688965 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.689728 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.690853 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.691163 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.691270 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.694253 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:11.694257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:11.694260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:11.694293 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:11.696058 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:11.696553 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:11.696556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:11.696558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:11.696672 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:11.696947 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:11.696951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:11.696953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:11.697184 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:11.697312 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.697553 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.697674 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:12:11.700889 11570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:11.701362 11570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:11.706434 11707 tablet_service.cc:1467] Processing CreateTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a (DEFAULT_TABLE table=test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.706781 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f7e1edaa574a4b9febda84b08dac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.706875 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a (DEFAULT_TABLE table=test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.707145 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f7e1edaa574a4b9febda84b08dac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.708933 11987 tablet_service.cc:1467] Processing CreateTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a (DEFAULT_TABLE table=test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.709192 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f7e1edaa574a4b9febda84b08dac1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.709419 12083 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Bootstrap starting.
I20260620 07:12:11.709460 12082 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Bootstrap starting.
I20260620 07:12:11.710227 12082 tablet_bootstrap.cc:654] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.710312 12083 tablet_bootstrap.cc:654] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.711467 12083 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: No bootstrap required, opened a new log
I20260620 07:12:11.711479 12082 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: No bootstrap required, opened a new log
I20260620 07:12:11.711544 12083 ts_tablet_manager.cc:1397] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:11.711565 12082 ts_tablet_manager.cc:1397] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:11.711628 12081 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Bootstrap starting.
I20260620 07:12:11.711943 12082 raft_consensus.cc:348] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.712046 12082 raft_consensus.cc:374] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.711969 12083 raft_consensus.cc:348] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.712078 12082 raft_consensus.cc:729] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.712088 12083 raft_consensus.cc:374] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.712121 12083 raft_consensus.cc:729] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.712186 12082 consensus_queue.cc:260] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.712249 12083 consensus_queue.cc:260] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.712435 12082 ts_tablet_manager.cc:1428] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.712529 12083 ts_tablet_manager.cc:1428] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.712551 12081 tablet_bootstrap.cc:654] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.715751 12081 tablet_bootstrap.cc:492] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: No bootstrap required, opened a new log
I20260620 07:12:11.715863 12081 ts_tablet_manager.cc:1397] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 07:12:11.716287 12081 raft_consensus.cc:348] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.716416 12081 raft_consensus.cc:374] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.716485 12081 raft_consensus.cc:729] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e970aaf9b774a03b950c1be543d3db5, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.716650 12081 consensus_queue.cc:260] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.716933 12081 ts_tablet_manager.cc:1428] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.805397 12135 raft_consensus.cc:482] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:11.805562 12135 raft_consensus.cc:504] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.806026 12135 leader_election.cc:288] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.806527 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f7e1edaa574a4b9febda84b08dac1a" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d" is_pre_election: true
I20260620 07:12:11.806565 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f7e1edaa574a4b9febda84b08dac1a" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5" is_pre_election: true
I20260620 07:12:11.806686 11727 raft_consensus.cc:2454] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:11.806746 12007 raft_consensus.cc:2454] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:11.806980 11801 leader_election.cc:302] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.807147 12135 raft_consensus.cc:2781] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:11.807195 12135 raft_consensus.cc:482] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:11.807235 12135 raft_consensus.cc:3037] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.808395 12135 raft_consensus.cc:504] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.808811 12135 leader_election.cc:288] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Requested vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:11.809242 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f7e1edaa574a4b9febda84b08dac1a" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d"
I20260620 07:12:11.809268 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f7e1edaa574a4b9febda84b08dac1a" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5"
I20260620 07:12:11.809381 12007 raft_consensus.cc:3037] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.809381 11727 raft_consensus.cc:3037] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:11.810518 12007 raft_consensus.cc:2454] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:11.810542 11727 raft_consensus.cc:2454] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:11.810868 11798 leader_election.cc:302] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e970aaf9b774a03b950c1be543d3db5, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:11.811066 12135 raft_consensus.cc:2781] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:11.811147 12135 raft_consensus.cc:686] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Becoming Leader. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Running, Role: LEADER
I20260620 07:12:11.811269 12135 consensus_queue.cc:237] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.813215 11569 catalog_manager.cc:5495] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2). New cstate: current_term: 1 leader_uuid: "e9a21cc577714a8fb414f2256bf0b20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:11.844985 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:11.841601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.3.245.2:43533)
0620 07:12:11.841768 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:11.841772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:11.841787 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:11.841919 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:11.841922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:11.841975 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:11.842066 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:11.842320 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.842939 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.843809 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:11.844041 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:11.844176 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:11.844513 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:11.844580 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:11.844669 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:11.844823 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 07:12:11.844871 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:12:11.847021 12007 raft_consensus.cc:1261] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.847040 11727 raft_consensus.cc:1261] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:11.847537 12113 consensus_queue.cc:1035] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.847721 12135 consensus_queue.cc:1035] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:11.854910 11569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.855091 11569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:11.857443 11569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Sending DeleteTablet for 3 replicas of tablet 94f7e1edaa574a4b9febda84b08dac1a
I20260620 07:12:11.858011 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:50172
I20260620 07:12:11.858137 11706 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260620 07:12:11.858224 12107 tablet_replica.cc:330] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: stopping tablet replica
I20260620 07:12:11.858290 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60194
I20260620 07:12:11.858333 12105 tablet_replica.cc:330] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: stopping tablet replica
I20260620 07:12:11.858355 12107 raft_consensus.cc:2229] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:11.858134 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f7e1edaa574a4b9febda84b08dac1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:11 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260620 07:12:11.858438 12105 raft_consensus.cc:2229] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.858474 12106 tablet_replica.cc:330] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: stopping tablet replica
I20260620 07:12:11.858533 12107 raft_consensus.cc:2258] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.858562 12105 raft_consensus.cc:2258] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.858587 12106 raft_consensus.cc:2229] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:11.858703 12106 raft_consensus.cc:2258] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:11.859375 12106 ts_tablet_manager.cc:1905] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.859380 12105 ts_tablet_manager.cc:1905] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.859786 12107 ts_tablet_manager.cc:1905] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:11.861383 12106 ts_tablet_manager.cc:1918] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.861440 12106 log.cc:1196] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/94f7e1edaa574a4b9febda84b08dac1a
I20260620 07:12:11.861651 12106 ts_tablet_manager.cc:1939] T 94f7e1edaa574a4b9febda84b08dac1a P 3e970aaf9b774a03b950c1be543d3db5: Deleting consensus metadata
I20260620 07:12:11.861821 12105 ts_tablet_manager.cc:1918] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.861889 12105 log.cc:1196] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/94f7e1edaa574a4b9febda84b08dac1a
I20260620 07:12:11.862151 12105 ts_tablet_manager.cc:1939] T 94f7e1edaa574a4b9febda84b08dac1a P 2957274c90674e78b0c35a7b03406b9d: Deleting consensus metadata
I20260620 07:12:11.862339 12107 ts_tablet_manager.cc:1918] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:11.862399 12107 log.cc:1196] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/94f7e1edaa574a4b9febda84b08dac1a
I20260620 07:12:11.862427 11554 catalog_manager.cc:4841] TS 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753): tablet 94f7e1edaa574a4b9febda84b08dac1a (table test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]) successfully deleted
I20260620 07:12:11.862666 12107 ts_tablet_manager.cc:1939] T 94f7e1edaa574a4b9febda84b08dac1a P e9a21cc577714a8fb414f2256bf0b20d: Deleting consensus metadata
I20260620 07:12:11.862841 11557 catalog_manager.cc:4841] TS 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389): tablet 94f7e1edaa574a4b9febda84b08dac1a (table test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]) successfully deleted
W20260620 07:12:11.863273 11557 catalog_manager.cc:4808] TS 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389): delete failed for tablet 94f7e1edaa574a4b9febda84b08dac1a because the tablet was not found. No further retry: Not found: Tablet not found: 94f7e1edaa574a4b9febda84b08dac1a
I20260620 07:12:11.863270 11554 catalog_manager.cc:4841] TS e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533): tablet 94f7e1edaa574a4b9febda84b08dac1a (table test-table [id=4bb6c9c1fb70403f89f6d8f72d748a0d]) successfully deleted
I20260620 07:12:11.863457 11569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:11.863932 11569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:11.868719 11707 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3c5df300b349fca05e2059faa08987 (DEFAULT_TABLE table=test-table [id=bead139580aa4b0a8aa5e9875045f4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.868885 11847 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3c5df300b349fca05e2059faa08987 (DEFAULT_TABLE table=test-table [id=bead139580aa4b0a8aa5e9875045f4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.869017 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3c5df300b349fca05e2059faa08987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.869179 11847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3c5df300b349fca05e2059faa08987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.871259 12082 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: Bootstrap starting.
I20260620 07:12:11.871383 12083 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Bootstrap starting.
I20260620 07:12:11.871627 11987 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3c5df300b349fca05e2059faa08987 (DEFAULT_TABLE table=test-table [id=bead139580aa4b0a8aa5e9875045f4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:11.871901 11987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e3c5df300b349fca05e2059faa08987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:11.872113 12082 tablet_bootstrap.cc:654] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.872114 12083 tablet_bootstrap.cc:654] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.873279 12083 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: No bootstrap required, opened a new log
I20260620 07:12:11.873365 12083 ts_tablet_manager.cc:1397] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:11.873678 12082 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: No bootstrap required, opened a new log
I20260620 07:12:11.873767 12082 ts_tablet_manager.cc:1397] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:11.873819 12083 raft_consensus.cc:348] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.873930 12083 raft_consensus.cc:374] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.873970 12083 raft_consensus.cc:729] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.874087 12083 consensus_queue.cc:260] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.874177 12081 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Bootstrap starting.
I20260620 07:12:11.874182 12082 raft_consensus.cc:348] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.874279 12082 raft_consensus.cc:374] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.874310 12082 raft_consensus.cc:729] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957274c90674e78b0c35a7b03406b9d, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.874363 12083 ts_tablet_manager.cc:1428] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:11.874419 12082 consensus_queue.cc:260] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.874666 12082 ts_tablet_manager.cc:1428] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 07:12:11.875075 12081 tablet_bootstrap.cc:654] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:11.876361 12081 tablet_bootstrap.cc:492] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: No bootstrap required, opened a new log
I20260620 07:12:11.876466 12081 ts_tablet_manager.cc:1397] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 07:12:11.876928 12081 raft_consensus.cc:348] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.877058 12081 raft_consensus.cc:374] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:11.877099 12081 raft_consensus.cc:729] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e970aaf9b774a03b950c1be543d3db5, State: Initialized, Role: FOLLOWER
I20260620 07:12:11.877213 12081 consensus_queue.cc:260] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:11.877497 12081 ts_tablet_manager.cc:1428] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:12:12.086061 12135 raft_consensus.cc:482] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:12.086225 12135 raft_consensus.cc:504] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:12.086966 12135 leader_election.cc:288] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:12.087394 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3c5df300b349fca05e2059faa08987" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d" is_pre_election: true
I20260620 07:12:12.087469 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3c5df300b349fca05e2059faa08987" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5" is_pre_election: true
I20260620 07:12:12.087574 11727 raft_consensus.cc:2454] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:12.087594 12007 raft_consensus.cc:2454] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 0.
I20260620 07:12:12.087927 11801 leader_election.cc:302] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2957274c90674e78b0c35a7b03406b9d, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:12.088204 12135 raft_consensus.cc:2781] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:12.088272 12135 raft_consensus.cc:482] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:12.088325 12135 raft_consensus.cc:3037] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:12.089478 12135 raft_consensus.cc:504] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:12.089906 12135 leader_election.cc:288] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Requested vote from peers 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389), 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753)
I20260620 07:12:12.090297 11727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3c5df300b349fca05e2059faa08987" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957274c90674e78b0c35a7b03406b9d"
I20260620 07:12:12.090358 12007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3c5df300b349fca05e2059faa08987" candidate_uuid: "e9a21cc577714a8fb414f2256bf0b20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e970aaf9b774a03b950c1be543d3db5"
I20260620 07:12:12.090436 11727 raft_consensus.cc:3037] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:12.090520 12007 raft_consensus.cc:3037] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:12.091584 12007 raft_consensus.cc:2454] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:12.091672 11727 raft_consensus.cc:2454] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a21cc577714a8fb414f2256bf0b20d in term 1.
I20260620 07:12:12.091977 11798 leader_election.cc:302] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e970aaf9b774a03b950c1be543d3db5, e9a21cc577714a8fb414f2256bf0b20d; no voters:
I20260620 07:12:12.092200 12135 raft_consensus.cc:2781] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:12.092355 12135 raft_consensus.cc:686] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Becoming Leader. State: Replica: e9a21cc577714a8fb414f2256bf0b20d, State: Running, Role: LEADER
I20260620 07:12:12.092512 12135 consensus_queue.cc:237] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } }
I20260620 07:12:12.094503 11569 catalog_manager.cc:5495] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2). New cstate: current_term: 1 leader_uuid: "e9a21cc577714a8fb414f2256bf0b20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a21cc577714a8fb414f2256bf0b20d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:12.123631 11727 raft_consensus.cc:1261] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:12.123656 12007 raft_consensus.cc:1261] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Refusing update from remote peer e9a21cc577714a8fb414f2256bf0b20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:12.124009 12135 consensus_queue.cc:1035] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2957274c90674e78b0c35a7b03406b9d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:12.124163 12087 consensus_queue.cc:1035] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e970aaf9b774a03b950c1be543d3db5" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:12.156970 11570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:12.157116 11570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:12.159106 11570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11644617fef44e2892b5841af6860378: Sending DeleteTablet for 3 replicas of tablet 9e3c5df300b349fca05e2059faa08987
I20260620 07:12:12.159734 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3c5df300b349fca05e2059faa08987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:12 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260620 07:12:12.159942 12105 tablet_replica.cc:330] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d: stopping tablet replica
I20260620 07:12:12.159914 11847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3c5df300b349fca05e2059faa08987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:12 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:50172
I20260620 07:12:12.160110 12105 raft_consensus.cc:2229] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:12.160189 12107 tablet_replica.cc:330] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: stopping tablet replica
I20260620 07:12:12.160154 11987 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e3c5df300b349fca05e2059faa08987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:12 UTC) from {username='oryx', principal='oryx/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60194
I20260620 07:12:12.160243 12105 raft_consensus.cc:2258] T 9e3c5df300b349fca05e2059faa08987 P 2957274c90674e78b0c35a7b03406b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:12.160298 12107 raft_consensus.cc:2229] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:12.160324 12106 tablet_replica.cc:330] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: stopping tablet replica
I20260620 07:12:12.160414 12106 raft_consensus.cc:2229] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:12.160513 12106 raft_consensus.cc:2258] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:12.160512 12107 raft_consensus.cc:2258] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:12.160712 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11642
I20260620 07:12:12.162375 12107 ts_tablet_manager.cc:1905] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:12.162593 12106 ts_tablet_manager.cc:1905] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:12.164710 12107 ts_tablet_manager.cc:1918] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:12:12.164803 12107 log.cc:1196] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/9e3c5df300b349fca05e2059faa08987
I20260620 07:12:12.165048 12107 ts_tablet_manager.cc:1939] T 9e3c5df300b349fca05e2059faa08987 P e9a21cc577714a8fb414f2256bf0b20d: Deleting consensus metadata
I20260620 07:12:12.165715 12106 ts_tablet_manager.cc:1918] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 07:12:12.165803 12106 log.cc:1196] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/9e3c5df300b349fca05e2059faa08987
I20260620 07:12:12.165752 11554 catalog_manager.cc:4841] TS e9a21cc577714a8fb414f2256bf0b20d (127.3.245.2:43533): tablet 9e3c5df300b349fca05e2059faa08987 (table test-table [id=bead139580aa4b0a8aa5e9875045f4b9]) successfully deleted
I20260620 07:12:12.166074 12106 ts_tablet_manager.cc:1939] T 9e3c5df300b349fca05e2059faa08987 P 3e970aaf9b774a03b950c1be543d3db5: Deleting consensus metadata
I20260620 07:12:12.166795 11554 catalog_manager.cc:4841] TS 3e970aaf9b774a03b950c1be543d3db5 (127.3.245.3:35753): tablet 9e3c5df300b349fca05e2059faa08987 (table test-table [id=bead139580aa4b0a8aa5e9875045f4b9]) successfully deleted
W20260620 07:12:12.169847 11557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:44389 (error 108)
I20260620 07:12:12.170073 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11782
I20260620 07:12:12.172402 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:12.171960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:44389 (local address 127.0.0.1:35544)
0620 07:12:12.172220 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:12.172289 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
W20260620 07:12:12.172566 11557 catalog_manager.cc:4584] TS 2957274c90674e78b0c35a7b03406b9d (127.3.245.1:44389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e3c5df300b349fca05e2059faa08987: Network error: Client connection negotiation failed: client connection to 127.3.245.1:44389: connect: Connection refused (error 111)
I20260620 07:12:12.177911 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11922
I20260620 07:12:12.183868 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 11538
2026-06-20T07:12:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3575 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:12 dist-test-slave-ldfr krb5kdc[12171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:12 dist-test-slave-ldfr krb5kdc[12171](info): set up 2 sockets
Jun 20 07:12:12 dist-test-slave-ldfr krb5kdc[12171](info): commencing operation
krb5kdc: starting...
W20260620 07:12:13.229877 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:13 dist-test-slave-ldfr krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:12:13.360543 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44935
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:44935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:13.475534 12184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:13.475813 12184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:13.475893 12184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:13.480131 12184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:13.480212 12184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:13.480233 12184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:13.480250 12184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:13.480270 12184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:13.485764 12184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:44935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:13.486997 12184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:13.488071 12184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:12:13.496969 12184 server_base.cc:1034] running on GCE node
W20260620 07:12:13.496990 12190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:13.496974 12192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:13.496990 12189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:13.497857 12184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:13.499107 12184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:13.500316 12184 hybrid_clock.cc:648] HybridClock initialized: now 1781939533500276 us; error 54 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T07:12:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1334 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:13 dist-test-slave-ldfr krb5kdc[12197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:13 dist-test-slave-ldfr krb5kdc[12197](info): set up 2 sockets
Jun 20 07:12:13 dist-test-slave-ldfr krb5kdc[12197](info): commencing operation
krb5kdc: starting...
W20260620 07:12:14.562152 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:14 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:12:14.685526 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46131
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:14.799329 12210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:14.799661 12210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:14.799741 12210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:14.804016 12210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:14.804116 12210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:14.804154 12210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:14.804196 12210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:14.804232 12210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:14.809695 12210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:14.810820 12210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:14.811905 12210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:14.820669 12215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:14.821053 12210 server_base.cc:1034] running on GCE node
W20260620 07:12:14.820659 12218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:14.820716 12216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:14.821775 12210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:14.822890 12210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:14.824071 12210 hybrid_clock.cc:648] HybridClock initialized: now 1781939534824051 us; error 32 us; skew 500 ppm
Jun 20 07:12:14 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:14.827335 12210 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:12:14.828357 12210 webserver.cc:460] Webserver started at http://127.3.245.62:43001/ using document root <none> and password file <none>
I20260620 07:12:14.829035 12210 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:14.829100 12210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:14.829332 12210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:14.831461 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "19c39f0d88a34bf1a22e9099b70c7f37"
format_stamp: "Formatted at 2026-06-20 07:12:14 on dist-test-slave-ldfr"
server_key: "93f1067fde8ec263958eadec1b310c5e"
server_key_iv: "370bd2605ad29bd02def27abda54318c"
server_key_version: "encryptionkey@0"
I20260620 07:12:14.832048 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "19c39f0d88a34bf1a22e9099b70c7f37"
format_stamp: "Formatted at 2026-06-20 07:12:14 on dist-test-slave-ldfr"
server_key: "93f1067fde8ec263958eadec1b310c5e"
server_key_iv: "370bd2605ad29bd02def27abda54318c"
server_key_version: "encryptionkey@0"
I20260620 07:12:14.836021 12210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 07:12:14.838770 12225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:14.840225 12210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:14.840394 12210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "19c39f0d88a34bf1a22e9099b70c7f37"
format_stamp: "Formatted at 2026-06-20 07:12:14 on dist-test-slave-ldfr"
server_key: "93f1067fde8ec263958eadec1b310c5e"
server_key_iv: "370bd2605ad29bd02def27abda54318c"
server_key_version: "encryptionkey@0"
I20260620 07:12:14.840519 12210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:14.862816 12210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:14.866590 12210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:14.866833 12210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:14.874985 12210 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:46131
I20260620 07:12:14.875018 12277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:46131 every 8 connection(s)
I20260620 07:12:14.875999 12210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:14.878962 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:14.882310 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12210
I20260620 07:12:14.882419 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:14.882706 4052 external_mini_cluster.cc:1428] Setting key b9db2c55f4a4e849bfa487c6311b2674
I20260620 07:12:14.885191 12278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Bootstrap starting.
I20260620 07:12:14.887465 12278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:14.888247 12278 log.cc:824] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:14.890254 12278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: No bootstrap required, opened a new log
Jun 20 07:12:14 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:14.894577 12278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:14.894822 12278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:14.894897 12278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c39f0d88a34bf1a22e9099b70c7f37, State: Initialized, Role: FOLLOWER
I20260620 07:12:14.895473 12278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:14.895691 12278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:14.895771 12278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:14.895941 12278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:14.897243 12278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:14.897631 12278 leader_election.cc:302] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c39f0d88a34bf1a22e9099b70c7f37; no voters:
I20260620 07:12:14.897957 12278 leader_election.cc:288] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:14.898068 12283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:14.898285 12283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 LEADER]: Becoming Leader. State: Replica: 19c39f0d88a34bf1a22e9099b70c7f37, State: Running, Role: LEADER
I20260620 07:12:14.898506 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:14.884204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.3.245.62:46131)
0620 07:12:14.884728 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:14.884745 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:14.884782 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:12:14.885559 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:14.885578 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:14.885942 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:14.886287 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:14.887099 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:14.888154 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:14.889147 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:14.889614 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:14.889795 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:14.892606 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:14.892656 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:14.892681 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:14.892716 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:14.894757 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:14.895374 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:14.895380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:14.895385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:14.895526 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:14.895847 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:14.895853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:14.895855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:14.896346 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:14.896563 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:14.896952 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 07:12:14.897394 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260620 07:12:14.898725 12283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:14.899387 12278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:14.900554 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c39f0d88a34bf1a22e9099b70c7f37. Latest consensus state: current_term: 1 leader_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } } }
I20260620 07:12:14.900804 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:14.901224 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } } }
I20260620 07:12:14.901402 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: This master's current role is: LEADER
W20260620 07:12:14.904412 12298 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:12:14.904512 12298 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:12:14.904914 12292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:14.905681 12292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:14.911500 12292 catalog_manager.cc:1347] Generated new cluster ID: 0514b4abaa614c7b9b1e75a839312e15
I20260620 07:12:14.911592 12292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:14.920179 12292 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:14.921329 12292 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:14.929003 12292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Generated new TSK 0
I20260620 07:12:14.929929 12292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:12:14.960341 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:15.078112 12304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:15.078452 12304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:15.078531 12304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:15.082839 12304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:15.082962 12304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:15.083060 12304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:15.088663 12304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:15.089829 12304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:15.090957 12304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:15.102336 12309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.102452 12310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.102586 12312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:15.105422 12304 server_base.cc:1034] running on GCE node
I20260620 07:12:15.106051 12304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:15.107066 12304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:15.108265 12304 hybrid_clock.cc:648] HybridClock initialized: now 1781939535108243 us; error 46 us; skew 500 ppm
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:15.111529 12304 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:12:15.112462 12304 webserver.cc:460] Webserver started at http://127.3.245.1:45227/ using document root <none> and password file <none>
I20260620 07:12:15.113121 12304 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:15.113195 12304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:15.113379 12304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:15.115597 12304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "d070aafb41964f0aacbcce7a8bb0ea74"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "4581fb4e18449692b0396cab5843d505"
server_key_iv: "fa048d9cb4ad23787af5cdbb5881719a"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.116149 12304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "d070aafb41964f0aacbcce7a8bb0ea74"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "4581fb4e18449692b0396cab5843d505"
server_key_iv: "fa048d9cb4ad23787af5cdbb5881719a"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.120814 12304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260620 07:12:15.123497 12319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.125063 12304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:15.125217 12304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "d070aafb41964f0aacbcce7a8bb0ea74"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "4581fb4e18449692b0396cab5843d505"
server_key_iv: "fa048d9cb4ad23787af5cdbb5881719a"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.125360 12304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:15.143446 12304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:15.147145 12304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:15.147414 12304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:15.148059 12304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:15.149037 12304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:15.149115 12304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.149196 12304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:15.149246 12304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.159584 12304 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37081
I20260620 07:12:15.159657 12432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37081 every 8 connection(s)
I20260620 07:12:15.160590 12304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:15.168059 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12304
I20260620 07:12:15.168200 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:15.168514 4052 external_mini_cluster.cc:1428] Setting key 6fabd164326ebcb89a1346817269ff2f
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:15.178910 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.162890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:48221)
0620 07:12:15.163349 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:15.163391 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:15.164599 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:15.165113 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:15.165119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:15.165484 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:15.165963 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.165976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.166797 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.166800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.167710 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.167720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.168356 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.168362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.168512 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.169234 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:15.169250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:15.170676 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:15.174624 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.174631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.174696 (+ 65us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.175208 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.175619 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.175623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.175625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.175855 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.176455 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:15.176463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:15.176967 (+ 504us) client_negotiation.cc:769] Sending connection context
0620 07:12:15.177236 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 07:12:15.177664 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:12:15.179728 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.162482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48221 (local address 127.3.245.62:46131)
0620 07:12:15.162620 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:15.162624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:15.163510 (+ 886us) server_negotiation.cc:407] Connection header received
0620 07:12:15.164849 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:15.164852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:15.164900 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:15.164978 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:15.166105 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.166674 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.167829 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.168181 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.168285 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.170844 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:15.170849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:15.170852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:15.170886 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:15.174472 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.175384 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.175388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.175390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.175497 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.175961 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.175966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.175969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.176296 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:15.176426 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:15.179393 (+ 2967us) server_negotiation.cc:299] Negotiation successful
0620 07:12:15.179597 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:12:15.180979 12433 heartbeater.cc:347] Connected to a master server at 127.3.245.62:46131
I20260620 07:12:15.181272 12433 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:15.181928 12433 heartbeater.cc:511] Master 127.3.245.62:46131 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:12:15.183694 12242 ts_manager.cc:194] Registered new tserver with Master: d070aafb41964f0aacbcce7a8bb0ea74 (127.3.245.1:37081)
I20260620 07:12:15.185046 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:48221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:12:15.199920 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:15.315816 12438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:15.316133 12438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:15.316219 12438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:15.320407 12438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:15.320515 12438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:15.320627 12438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:15.325989 12438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:15.327126 12438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:15.328192 12438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:15.338297 12446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.338295 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.338467 12444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:15.338634 12438 server_base.cc:1034] running on GCE node
I20260620 07:12:15.339162 12438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:15.339866 12438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:15.341156 12438 hybrid_clock.cc:648] HybridClock initialized: now 1781939535341133 us; error 40 us; skew 500 ppm
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:15.344208 12438 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:12:15.345321 12438 webserver.cc:460] Webserver started at http://127.3.245.2:34111/ using document root <none> and password file <none>
I20260620 07:12:15.345937 12438 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:15.346028 12438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:15.346246 12438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:15.348361 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "b7f1c2cfc18c4d23a8284e9930c8117d"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "2d7ec5398a5f0617eb9baed7155053db"
server_key_iv: "2cb9ebd5b5a0310522ff473124d6c0bc"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.348981 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "b7f1c2cfc18c4d23a8284e9930c8117d"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "2d7ec5398a5f0617eb9baed7155053db"
server_key_iv: "2cb9ebd5b5a0310522ff473124d6c0bc"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.352878 12438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:15.355494 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.356933 12438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:15.357085 12438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "b7f1c2cfc18c4d23a8284e9930c8117d"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "2d7ec5398a5f0617eb9baed7155053db"
server_key_iv: "2cb9ebd5b5a0310522ff473124d6c0bc"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.357209 12438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:15.367113 12438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:15.370400 12438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:15.370611 12438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:15.371212 12438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:15.372061 12438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:15.372130 12438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.372197 12438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:15.372243 12438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.381274 12438 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:36587
I20260620 07:12:15.381310 12566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:36587 every 8 connection(s)
I20260620 07:12:15.382185 12438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:15.387059 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12438
I20260620 07:12:15.387202 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:15.387549 4052 external_mini_cluster.cc:1428] Setting key 0754ef13a0752c3dc1b184fd3f7a79f1
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:12:15.397516 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.384106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:38601 (local address 127.3.245.62:46131)
0620 07:12:15.384268 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:15.384271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:15.385131 (+ 860us) server_negotiation.cc:407] Connection header received
0620 07:12:15.386394 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:15.386397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:15.386448 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:15.386543 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:15.388229 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.388959 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.390717 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.390984 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.391110 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.393663 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:15.393667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:15.393669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:15.393706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:15.395185 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.395744 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.395750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.395751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.395837 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.396199 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.396203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.396205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.396402 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:15.396532 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:15.397187 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 07:12:15.397334 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:12:15.398598 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.384453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:38601)
0620 07:12:15.384970 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:15.385013 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:15.386154 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:15.386723 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:15.386752 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:15.387346 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:15.388041 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.388059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.389114 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.389116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.390575 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.390590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.391152 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.391156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.391329 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.391993 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:15.392011 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:15.393480 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:15.395317 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.395324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.395342 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.395634 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.395933 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.395938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.395940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.396099 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.396531 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:15.396539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:15.396938 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 07:12:15.397148 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 07:12:15.397500 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:12:15.400240 12567 heartbeater.cc:347] Connected to a master server at 127.3.245.62:46131
I20260620 07:12:15.400521 12567 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:15.401129 12567 heartbeater.cc:511] Master 127.3.245.62:46131 requested a full tablet report, sending...
I20260620 07:12:15.402271 12242 ts_manager.cc:194] Registered new tserver with Master: b7f1c2cfc18c4d23a8284e9930c8117d (127.3.245.2:36587)
I20260620 07:12:15.402861 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:38601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:12:15.420447 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:15.534335 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:15.534596 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:15.534641 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:15.538899 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:15.538962 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:15.539049 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:15.544445 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:15.545639 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:15.546765 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:15.556507 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.556519 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.556511 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:15.556753 12572 server_base.cc:1034] running on GCE node
I20260620 07:12:15.557219 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:15.557868 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:15.559038 12572 hybrid_clock.cc:648] HybridClock initialized: now 1781939535559023 us; error 26 us; skew 500 ppm
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:15.561980 12572 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:12:15.562873 12572 webserver.cc:460] Webserver started at http://127.3.245.3:43949/ using document root <none> and password file <none>
I20260620 07:12:15.563516 12572 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:15.563614 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:15.563907 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:15.566617 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "6afa3cb7e38044cc9ed9ccee2f22e3d6"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "f5836735b2d9812096b913c97d651dc7"
server_key_iv: "18f40729452efcc0e6f8f8e57e4cda2f"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.567207 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "6afa3cb7e38044cc9ed9ccee2f22e3d6"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "f5836735b2d9812096b913c97d651dc7"
server_key_iv: "18f40729452efcc0e6f8f8e57e4cda2f"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.571182 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:12:15.573913 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.575336 12572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:15.575510 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "6afa3cb7e38044cc9ed9ccee2f22e3d6"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "f5836735b2d9812096b913c97d651dc7"
server_key_iv: "18f40729452efcc0e6f8f8e57e4cda2f"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.575639 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:15.595587 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:15.599334 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:15.599599 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:15.600329 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:15.601318 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:15.601390 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.601454 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:15.601487 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.611351 12572 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:43447
I20260620 07:12:15.611375 12700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:43447 every 8 connection(s)
I20260620 07:12:15.612311 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:15.617419 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12572
I20260620 07:12:15.617542 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:15.617851 4052 external_mini_cluster.cc:1428] Setting key dfa94d1f98f3ab0abc9339e3574f37ed
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:15.627542 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.614439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:43127 (local address 127.3.245.62:46131)
0620 07:12:15.614588 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:15.614592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:15.615376 (+ 784us) server_negotiation.cc:407] Connection header received
0620 07:12:15.616673 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:15.616680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:15.616746 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:15.616828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:15.618598 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.619259 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.620881 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.621148 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.621245 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.624014 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:15.624018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:15.624020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:15.624047 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:15.625596 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.626120 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.626123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.626125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.626207 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:15.626551 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:15.626554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:15.626555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:15.626739 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:15.626819 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:15.627299 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 07:12:15.627410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:12:15.628461 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:15.614703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:43127)
0620 07:12:15.615202 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:15.615246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:15.616392 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:15.617042 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:15.617050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:15.617450 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:15.618454 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.618473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.619521 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.619525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.620744 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:15.620754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:15.621322 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:15.621325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:15.621501 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:15.622140 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:15.622157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:15.623838 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:15.625714 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.625719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.625734 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.626003 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.626293 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:15.626295 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:15.626296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:15.626458 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:15.626826 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:15.626831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:15.627088 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 07:12:15.627251 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 07:12:15.627555 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260620 07:12:15.629809 12701 heartbeater.cc:347] Connected to a master server at 127.3.245.62:46131
I20260620 07:12:15.630070 12701 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:15.630597 12701 heartbeater.cc:511] Master 127.3.245.62:46131 requested a full tablet report, sending...
I20260620 07:12:15.631783 12242 ts_manager.cc:194] Registered new tserver with Master: 6afa3cb7e38044cc9ed9ccee2f22e3d6 (127.3.245.3:43447)
I20260620 07:12:15.632347 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:43127
I20260620 07:12:15.644389 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
Entry for principal oryx/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab.
WARNING: no policy specified for oryx/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
Entry for principal oryx/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab.
WARNING: no policy specified for oryx/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
Entry for principal oryx/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab.
I20260620 07:12:15.719934 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12304
I20260620 07:12:15.726058 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12438
I20260620 07:12:15.731644 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12572
I20260620 07:12:15.737113 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12210
I20260620 07:12:15.743722 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46131
--webserver_interface=127.3.245.62
--webserver_port=43001
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:15.854743 12710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:15.855055 12710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:15.855129 12710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:15.859203 12710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:15.859314 12710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:15.859356 12710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:15.859400 12710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:15.859431 12710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:15.865269 12710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=43001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:15.866415 12710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:15.867518 12710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:15.876439 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:15.876502 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:15.876554 12710 server_base.cc:1034] running on GCE node
W20260620 07:12:15.876488 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:15.877331 12710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:15.878480 12710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:15.879662 12710 hybrid_clock.cc:648] HybridClock initialized: now 1781939535879646 us; error 29 us; skew 500 ppm
Jun 20 07:12:15 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939535, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:15.882555 12710 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:12:15.883534 12710 webserver.cc:460] Webserver started at http://127.3.245.62:43001/ using document root <none> and password file <none>
I20260620 07:12:15.884069 12710 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:15.884141 12710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:15.888265 12710 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:15.890928 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:15.892344 12710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:12:15.892503 12710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "19c39f0d88a34bf1a22e9099b70c7f37"
format_stamp: "Formatted at 2026-06-20 07:12:14 on dist-test-slave-ldfr"
server_key: "93f1067fde8ec263958eadec1b310c5e"
server_key_iv: "370bd2605ad29bd02def27abda54318c"
server_key_version: "encryptionkey@0"
I20260620 07:12:15.893216 12710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:15.905982 12710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:15.909519 12710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:15.909746 12710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:15.917390 12710 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:46131
I20260620 07:12:15.917419 12777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:46131 every 8 connection(s)
I20260620 07:12:15.918517 12710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:15.920688 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12710
I20260620 07:12:15.921178 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:37081
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:46131
--webserver_port=45227
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:12:15.923750 12778 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:12:15.925634 12778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Bootstrap starting.
I20260620 07:12:15.943135 12778 log.cc:824] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:15.949990 12778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:12:15.950393 12778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Bootstrap complete.
I20260620 07:12:15.955022 12778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:15.955495 12778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c39f0d88a34bf1a22e9099b70c7f37, State: Initialized, Role: FOLLOWER
I20260620 07:12:15.956060 12778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:15.956183 12778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:15.956266 12778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:15.956360 12778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:12:15.958381 12778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:15.958729 12778 leader_election.cc:302] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c39f0d88a34bf1a22e9099b70c7f37; no voters:
I20260620 07:12:15.959070 12778 leader_election.cc:288] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:12:15.959159 12782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:12:15.959352 12782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [term 2 LEADER]: Becoming Leader. State: Replica: 19c39f0d88a34bf1a22e9099b70c7f37, State: Running, Role: LEADER
I20260620 07:12:15.959661 12782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } }
I20260620 07:12:15.960354 12778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:15.960683 12782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c39f0d88a34bf1a22e9099b70c7f37. Latest consensus state: current_term: 2 leader_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } } }
I20260620 07:12:15.960810 12782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:15.961194 12783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c39f0d88a34bf1a22e9099b70c7f37" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 46131 } } }
I20260620 07:12:15.961376 12783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:15.962539 12786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:15.965801 12786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:15.966435 12786 catalog_manager.cc:1259] Loaded cluster ID: 0514b4abaa614c7b9b1e75a839312e15
I20260620 07:12:15.966526 12786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:15.969022 12786 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:15.969810 12786 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19c39f0d88a34bf1a22e9099b70c7f37: Loaded TSK: 0
I20260620 07:12:15.971122 12786 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:12:16.039499 12780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:16.039819 12780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:16.039896 12780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:16.044190 12780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:16.044308 12780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:16.044410 12780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:16.050164 12780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.1.keytab
--principal=oryx/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:37081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=45227
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:16.051326 12780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:16.052340 12780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:16.063797 12805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.063953 12807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.063953 12804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:16.065264 12780 server_base.cc:1034] running on GCE node
I20260620 07:12:16.065728 12780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:16.066483 12780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:16.067685 12780 hybrid_clock.cc:648] HybridClock initialized: now 1781939536067654 us; error 54 us; skew 500 ppm
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:16.070744 12780 init.cc:377] Logged in from keytab as oryx/127.3.245.1@KRBTEST.COM (short username oryx)
I20260620 07:12:16.071794 12780 webserver.cc:460] Webserver started at http://127.3.245.1:45227/ using document root <none> and password file <none>
I20260620 07:12:16.072515 12780 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:16.072669 12780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:16.076798 12780 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:12:16.079097 12814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.080446 12780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:16.080545 12780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "d070aafb41964f0aacbcce7a8bb0ea74"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "4581fb4e18449692b0396cab5843d505"
server_key_iv: "fa048d9cb4ad23787af5cdbb5881719a"
server_key_version: "encryptionkey@0"
I20260620 07:12:16.081259 12780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:16.107228 12780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:16.110885 12780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:16.111105 12780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:16.111793 12780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:16.112757 12780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:16.112807 12780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.112848 12780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:16.112864 12780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.122736 12780 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37081
I20260620 07:12:16.122761 12927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37081 every 8 connection(s)
I20260620 07:12:16.123811 12780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:16.128721 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12780
I20260620 07:12:16.129283 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:36587
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:46131
--webserver_port=34111
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:12:16.140527 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.126372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:34527 (local address 127.3.245.62:46131)
0620 07:12:16.126840 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.126859 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.127079 (+ 220us) server_negotiation.cc:407] Connection header received
0620 07:12:16.128734 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.128752 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.129207 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.129771 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.131088 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.132503 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.134077 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.134972 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.135188 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.139550 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:34527: BlockingRecv error: recv got EOF from 127.3.245.1:34527 (error 108)
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260620 07:12:16.140550 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.126383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:34527)
0620 07:12:16.126902 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.126957 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.128182 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.129976 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.129983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.130377 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.130888 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.130904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.132747 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.132751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.133892 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.133911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.135169 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.135172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.135350 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.136094 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.136115 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.139282 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
W20260620 07:12:16.141377 12928 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:46131 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.245.62:46131: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.148290 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.141973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:55963)
0620 07:12:16.142134 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.142151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.142261 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.142599 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.142602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.142755 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.142925 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.142931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.144005 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.144008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.144749 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.144758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.145400 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.145403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.145479 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.146045 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.146057 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.148170 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 07:12:16.148605 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.142169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:55963 (local address 127.3.245.62:46131)
0620 07:12:16.142318 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.142323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.142341 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:16.142388 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.142391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.142449 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.142535 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.143048 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.143875 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.144918 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.145271 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.145374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.148350 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:55963: BlockingRecv error: recv got EOF from 127.3.245.1:55963 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.155414 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.148939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:53071)
0620 07:12:16.149056 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.149072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.149160 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.149416 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.149418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.149545 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.149620 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.149625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.150669 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.150672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.151311 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.151321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.151960 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.151964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.152032 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.152597 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.152663 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.155296 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:12:16.155609 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.149032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:53071 (local address 127.3.245.62:46131)
0620 07:12:16.149153 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.149157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.149172 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:16.149260 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.149262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.149308 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.149387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.149719 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.150528 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.151464 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.151809 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.151915 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.155483 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:53071: BlockingRecv error: recv got EOF from 127.3.245.1:53071 (error 108)
Metrics: {"server-negotiator.queue_time_us":56,"thread_start_us":127,"threads_started":1}
W20260620 07:12:16.155747 12928 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:12:16.250648 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:16.250954 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:16.251000 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:16.255134 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:16.255205 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:16.255265 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:16.260958 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.2.keytab
--principal=oryx/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:36587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=34111
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:16.262127 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:16.263211 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:16.273429 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.273583 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.273583 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:16.273803 12932 server_base.cc:1034] running on GCE node
I20260620 07:12:16.274780 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:16.275576 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:16.276777 12932 hybrid_clock.cc:648] HybridClock initialized: now 1781939536276746 us; error 44 us; skew 500 ppm
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:16.279824 12932 init.cc:377] Logged in from keytab as oryx/127.3.245.2@KRBTEST.COM (short username oryx)
I20260620 07:12:16.280901 12932 webserver.cc:460] Webserver started at http://127.3.245.2:34111/ using document root <none> and password file <none>
I20260620 07:12:16.281571 12932 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:16.281687 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:16.286096 12932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:12:16.288499 12948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.290045 12932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:16.290207 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "b7f1c2cfc18c4d23a8284e9930c8117d"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "2d7ec5398a5f0617eb9baed7155053db"
server_key_iv: "2cb9ebd5b5a0310522ff473124d6c0bc"
server_key_version: "encryptionkey@0"
I20260620 07:12:16.290803 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:16.306372 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:16.309969 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:16.310182 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:16.310806 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:16.311666 12932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:16.311738 12932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.311825 12932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:16.311877 12932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.322166 12932 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:36587
I20260620 07:12:16.322209 13061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:36587 every 8 connection(s)
I20260620 07:12:16.323207 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:16.327749 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 12932
I20260620 07:12:16.328243 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:43447
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:46131
--webserver_port=43949
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:12:16.335983 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.325378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:55421 (local address 127.3.245.62:46131)
0620 07:12:16.325554 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.325558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.326393 (+ 835us) server_negotiation.cc:407] Connection header received
0620 07:12:16.327749 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.327753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.327818 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.327900 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.329333 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.329880 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.331400 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.331627 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.331723 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.335842 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:55421: BlockingRecv error: recv got EOF from 127.3.245.2:55421 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:12:16.336710 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.325761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:55421)
0620 07:12:16.326229 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.326284 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.327527 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.328081 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.328088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.328558 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.329158 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.329174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.330017 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.330021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.331251 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.331265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.331825 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.331832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.332085 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.332828 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.332858 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.335698 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
W20260620 07:12:16.337263 13062 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:46131 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.245.62:46131: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.343144 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.337792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:42315)
0620 07:12:16.337974 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.337991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.338120 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.338378 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.338381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.338560 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.338654 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.338659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.339390 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.339393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.339954 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.339962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.340439 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.340442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.340537 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.341100 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.341111 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.343039 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 07:12:16.343340 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.337856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:42315 (local address 127.3.245.62:46131)
0620 07:12:16.337973 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.337977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.338066 (+ 89us) server_negotiation.cc:407] Connection header received
0620 07:12:16.338221 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.338224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.338275 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.338362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.338770 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.339285 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.340085 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.340308 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.340432 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.343216 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:42315: BlockingRecv error: recv got EOF from 127.3.245.2:42315 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.349725 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.343974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:54041)
0620 07:12:16.344139 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.344152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.344226 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.344556 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.344559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.344777 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.344878 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.344885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.345613 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.345617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.346496 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.346507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.346919 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.346922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.347004 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.347487 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.347501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.349568 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 07:12:16.349818 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.344186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:54041 (local address 127.3.245.62:46131)
0620 07:12:16.344306 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.344309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.344321 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:12:16.344357 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.344359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.344423 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.344516 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.344993 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.345502 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.346614 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.346816 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.346899 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.349698 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:54041: BlockingRecv error: recv got EOF from 127.3.245.2:54041 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 07:12:16.350077 13062 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:12:16.446724 13065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:16.447021 13065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:16.447067 13065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:16.451227 13065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:16.451305 13065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:16.451393 13065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:16.457185 13065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/krb5kdc/oryx_127.3.245.3.keytab
--principal=oryx/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:43447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=43949
--tserver_master_addrs=127.3.245.62:46131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:16.458359 13065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:16.459532 13065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:16.470793 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.470782 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:16.470772 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:16.471268 13065 server_base.cc:1034] running on GCE node
I20260620 07:12:16.471777 13065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:16.472824 13065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:16.474149 13065 hybrid_clock.cc:648] HybridClock initialized: now 1781939536474138 us; error 75 us; skew 500 ppm
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939536, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:16.477492 13065 init.cc:377] Logged in from keytab as oryx/127.3.245.3@KRBTEST.COM (short username oryx)
I20260620 07:12:16.478516 13065 webserver.cc:460] Webserver started at http://127.3.245.3:43949/ using document root <none> and password file <none>
I20260620 07:12:16.479094 13065 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:16.479171 13065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:16.483721 13065 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:12:16.486528 13080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.488088 13065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:16.488209 13065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "6afa3cb7e38044cc9ed9ccee2f22e3d6"
format_stamp: "Formatted at 2026-06-20 07:12:15 on dist-test-slave-ldfr"
server_key: "f5836735b2d9812096b913c97d651dc7"
server_key_iv: "18f40729452efcc0e6f8f8e57e4cda2f"
server_key_version: "encryptionkey@0"
I20260620 07:12:16.488885 13065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:16.521164 13065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:16.524677 13065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:16.524858 13065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:16.525472 13065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:16.526341 13065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:16.526389 13065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.526428 13065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:16.526444 13065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:16.535367 13065 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:43447
I20260620 07:12:16.535391 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:43447 every 8 connection(s)
I20260620 07:12:16.536448 13065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:16.546046 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13065
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 07:12:16.548332 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.538430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:33335 (local address 127.3.245.62:46131)
0620 07:12:16.538573 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.538576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.539477 (+ 901us) server_negotiation.cc:407] Connection header received
0620 07:12:16.540771 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.540774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.540821 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.540896 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.542106 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.542622 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.543895 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.544138 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.544226 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.548210 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:33335: BlockingRecv error: recv got EOF from 127.3.245.3:33335 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:12:16.549099 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.538849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:33335)
0620 07:12:16.539293 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.539335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.540506 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.541047 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.541054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.541418 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.541960 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.541976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.542818 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.542821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.543764 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.543773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.544283 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.544285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.544442 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.545143 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.545161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.548103 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
W20260620 07:12:16.549998 13194 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:46131 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.245.62:46131: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.557266 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.547274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.3.245.62:46131)
0620 07:12:16.547472 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.547486 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.547502 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:12:16.547553 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.547556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.547606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.547711 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.548547 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.549140 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.550342 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.550614 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.550710 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.552416 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:16.552425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:16.552452 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:16.552494 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:16.554634 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:16.555282 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:16.555291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:16.555298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:16.555500 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:16.555804 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:16.555807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:16.555808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:16.556324 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:16.556583 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:16.556850 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:12:16.557113 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:12:16.557297 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.550511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:50879)
0620 07:12:16.550717 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.550733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.550826 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.551187 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.551189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.551420 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.551577 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.551586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.552545 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.552548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.553354 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.553364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.553979 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.553982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.554055 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.554530 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.554542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.557144 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 07:12:16.557384 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.550603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:50879 (local address 127.3.245.62:46131)
0620 07:12:16.550732 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.550736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.550779 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:12:16.550929 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.550932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.550980 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.551101 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.551722 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.552393 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.553486 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.553826 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.553949 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.557278 (+ 3329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:50879: BlockingRecv error: recv got EOF from 127.3.245.3:50879 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:16 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:16.564348 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:16.558240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:33787)
0620 07:12:16.558427 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:16.558441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:16.558523 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:16.558747 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:16.558750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:16.558939 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:16.559033 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.559039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.559878 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.559881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.560749 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:16.560761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.561501 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.561509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:16.561581 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.562184 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:16.562199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:16.564202 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260620 07:12:16.564467 12930 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:16.558403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:33787 (local address 127.3.245.62:46131)
0620 07:12:16.558538 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:16.558543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:16.558560 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:16.558598 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:16.558601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:16.558647 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:16.558731 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:16.559146 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.559740 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.561038 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:16.561357 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:16.561463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:16.564346 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:33787: BlockingRecv error: recv got EOF from 127.3.245.3:33787 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 07:12:16.564813 13194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:17.162166 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:17.156365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:58651)
0620 07:12:17.156681 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:17.156696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:17.156784 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:17.157074 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:17.157076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:17.157247 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:17.157344 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.157349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.158157 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.158159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.158742 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.158750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.159227 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.159229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.159292 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.159774 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:17.159785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:17.162019 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":96,"threads_started":1}
W20260620 07:12:17.162345 13199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:17.156472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:58651 (local address 127.3.245.62:46131)
0620 07:12:17.156743 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:17.156746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:17.156760 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:17.156917 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:17.156919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:17.156961 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:17.157039 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:17.157463 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.158047 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.158908 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.159128 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.159208 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.162190 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:58651: BlockingRecv error: recv got EOF from 127.3.245.1:58651 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":109,"threads_started":1}
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:17.356814 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:17.350771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:60025)
0620 07:12:17.351067 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:17.351080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:17.351182 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:17.351413 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:17.351415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:17.351741 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:17.351888 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.351896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.352720 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.352723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.353351 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.353360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.353867 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.353870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.353942 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.354497 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:17.354509 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:17.356599 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
W20260620 07:12:17.356982 13199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:17.350855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:60025 (local address 127.3.245.62:46131)
0620 07:12:17.351016 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:17.351021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:17.351135 (+ 114us) server_negotiation.cc:407] Connection header received
0620 07:12:17.351263 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:17.351265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:17.351313 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:17.351382 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:17.352019 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.352546 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.353488 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.353716 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.353807 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.356814 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:60025: BlockingRecv error: recv got EOF from 127.3.245.2:60025 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:17 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:17.571591 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:17.565425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:55103)
0620 07:12:17.565767 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:17.565783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:17.565928 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:17.566215 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:17.566218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:17.566476 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:17.566570 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.566576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.567408 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.567415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.568016 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:17.568024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.568480 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.568483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:17.568567 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.569148 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:17.569161 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:17.571408 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":86,"threads_started":1}
W20260620 07:12:17.571746 13199 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:17.565551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:55103 (local address 127.3.245.62:46131)
0620 07:12:17.565744 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:17.565749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:17.565874 (+ 125us) server_negotiation.cc:407] Connection header received
0620 07:12:17.566039 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:17.566043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:17.566096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:17.566188 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:17.566694 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.567288 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.568137 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:17.568368 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:17.568478 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:17.571595 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:55103: BlockingRecv error: recv got EOF from 127.3.245.3:55103 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:18.170421 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:18.163374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:53593)
0620 07:12:18.163769 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:18.163786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:18.163902 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:18.164269 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:18.164272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:18.164584 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:18.164730 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.164737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.165829 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.165871 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.166800 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.166814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.167315 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.167319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.167405 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.167994 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:18.168010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:18.170284 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":77,"threads_started":1}
W20260620 07:12:18.170893 13203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:18.163458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:53593 (local address 127.3.245.62:46131)
0620 07:12:18.163985 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:18.163991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:18.164008 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:12:18.164053 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:18.164055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:18.164110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:18.164220 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:18.164852 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.165721 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.166949 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.167183 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.167300 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.170506 (+ 3206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:53593: BlockingRecv error: recv got EOF from 127.3.245.1:53593 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":90,"threads_started":1}
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:18.363687 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:18.357654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:58221)
0620 07:12:18.357938 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:18.357958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:18.358056 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:18.358337 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:18.358340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:18.358522 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:18.358620 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.358629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.359496 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.359500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.360102 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.360111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.360692 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.360697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.360809 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.361381 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:18.361394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:18.363546 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
W20260620 07:12:18.363922 13203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:18.357780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:58221 (local address 127.3.245.62:46131)
0620 07:12:18.357938 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:18.357941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:18.358064 (+ 123us) server_negotiation.cc:407] Connection header received
0620 07:12:18.358119 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:18.358121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:18.358181 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:18.358270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:18.358739 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.359376 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.360233 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.360466 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.360597 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.363705 (+ 3108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:58221: BlockingRecv error: recv got EOF from 127.3.245.2:58221 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:18 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:18.584227 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:18.572472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:45227)
0620 07:12:18.572797 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:18.572813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:18.572927 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:18.573258 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:18.573261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:18.573458 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:18.573550 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.573556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.574366 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.574371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.575242 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:18.575253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.575972 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.575976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:18.576090 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.581833 (+ 5743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:18.581851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:18.584017 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":133,"threads_started":1}
W20260620 07:12:18.584275 13203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:18.572682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:45227 (local address 127.3.245.62:46131)
0620 07:12:18.572847 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:18.572854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:18.572870 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:12:18.573051 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:18.573055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:18.573110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:18.573197 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:18.573669 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.574202 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.575391 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:18.575819 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:18.575928 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:18.584156 (+ 8228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:45227: BlockingRecv error: recv got EOF from 127.3.245.3:45227 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:19.178990 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:19.171282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:36511)
0620 07:12:19.171545 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:19.171558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:19.171646 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:19.172061 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:19.172067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:19.172329 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:19.172473 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.172483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.173795 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.173802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.174901 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.174916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.175713 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.175717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.175802 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.176355 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:19.176371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:19.178793 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260620 07:12:19.179064 13207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:19.171382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:36511 (local address 127.3.245.62:46131)
0620 07:12:19.171645 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:19.171649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:19.171781 (+ 132us) server_negotiation.cc:407] Connection header received
0620 07:12:19.171830 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:19.171833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:19.171911 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:19.172025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:19.172645 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.173628 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.175089 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.175563 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.175683 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.178930 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:36511: BlockingRecv error: recv got EOF from 127.3.245.1:36511 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:19.370034 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:19.364515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:50563)
0620 07:12:19.364780 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:19.364795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:19.364909 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:19.365176 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:19.365179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:19.365368 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:19.365459 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.365465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.366260 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.366263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.366889 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.366898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.367370 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.367373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.367443 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.368031 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:19.368043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:19.369895 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":127,"threads_started":1}
W20260620 07:12:19.370203 13207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:19.364655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:50563 (local address 127.3.245.62:46131)
0620 07:12:19.364791 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:19.364796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:19.364880 (+ 84us) server_negotiation.cc:407] Connection header received
0620 07:12:19.364990 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:19.364992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:19.365035 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:19.365104 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:19.365599 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.366131 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.367032 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.367258 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.367347 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.370062 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:50563: BlockingRecv error: recv got EOF from 127.3.245.2:50563 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:19 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:19.591364 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:19.585159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:35521)
0620 07:12:19.585463 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:19.585485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:19.585619 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:19.585947 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:19.585950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:19.586165 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:19.586315 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.586326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.587224 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.587228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.587853 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:19.587862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.588318 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.588321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:19.588403 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.589266 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:19.589281 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:19.591174 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
W20260620 07:12:19.591423 13207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:19.585233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:35521 (local address 127.3.245.62:46131)
0620 07:12:19.585402 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:19.585407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:19.585561 (+ 154us) server_negotiation.cc:407] Connection header received
0620 07:12:19.585789 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:19.585792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:19.585851 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:19.585928 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:19.586434 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.587119 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.587970 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:19.588210 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:19.588316 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:19.591310 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:35521: BlockingRecv error: recv got EOF from 127.3.245.3:35521 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:20.186043 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:20.180038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:58751)
0620 07:12:20.180444 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:20.180459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:20.180577 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:20.180982 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:20.180986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:20.181198 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:20.181295 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.181301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.182093 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.182100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.182688 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.182696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.183208 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.183212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.183310 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.183821 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:20.183834 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:20.185926 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
W20260620 07:12:20.186360 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:20.180095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:58751 (local address 127.3.245.62:46131)
0620 07:12:20.180420 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:20.180425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:20.180531 (+ 106us) server_negotiation.cc:407] Connection header received
0620 07:12:20.180729 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:20.180732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:20.180801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:20.180901 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:20.181420 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.181974 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.182817 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.183069 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.183204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.186092 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:58751: BlockingRecv error: recv got EOF from 127.3.245.1:58751 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:20.377126 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:20.371143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:43405)
0620 07:12:20.371413 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:20.371429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:20.371535 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:20.371823 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:20.371826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:20.372029 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:20.372118 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.372124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.373137 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.373140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.373803 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.373813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.374266 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.374269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.374337 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.374965 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:20.374979 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:20.377007 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
W20260620 07:12:20.377419 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:20.371343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:43405 (local address 127.3.245.62:46131)
0620 07:12:20.371494 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:20.371498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:20.371513 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:20.371653 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:20.371657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:20.371708 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:20.371804 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:20.372238 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.372876 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.373936 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.374145 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.374226 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.377148 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:43405: BlockingRecv error: recv got EOF from 127.3.245.2:43405 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:20 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:20.598336 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:20.592202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:48141)
0620 07:12:20.592467 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:20.592482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:20.592593 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:20.592953 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:20.592956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:20.593162 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:20.593263 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.593268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.594201 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.594205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.594839 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:20.594848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.595346 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.595349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:20.595429 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.596000 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:20.596012 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:20.598226 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
W20260620 07:12:20.598589 13211 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:20.592315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:48141 (local address 127.3.245.62:46131)
0620 07:12:20.592448 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:20.592452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:20.592561 (+ 109us) server_negotiation.cc:407] Connection header received
0620 07:12:20.592771 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:20.592774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:20.592821 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:20.592914 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:20.593410 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.594085 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.594954 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:20.595233 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:20.595340 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:20.598364 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:48141: BlockingRecv error: recv got EOF from 127.3.245.3:48141 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:21.193017 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:21.186815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:38043)
0620 07:12:21.187188 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:21.187202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:21.187297 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:21.187559 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:21.187561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:21.187751 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:21.187844 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.187850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.188726 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.188729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.189306 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.189315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.189898 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.189901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.189972 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.190528 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:21.190540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:21.192901 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":82,"threads_started":1}
W20260620 07:12:21.193223 13215 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:21.186919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:38043 (local address 127.3.245.62:46131)
0620 07:12:21.187181 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:21.187184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:21.187252 (+ 68us) server_negotiation.cc:407] Connection header received
0620 07:12:21.187361 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:21.187363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:21.187402 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:21.187475 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:21.187956 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.188503 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.189443 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.189759 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.189861 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.193056 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:38043: BlockingRecv error: recv got EOF from 127.3.245.1:38043 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:21.384126 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:21.378054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:52027)
0620 07:12:21.378422 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:21.378436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:21.378565 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:21.378937 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:21.378940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:21.379122 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:21.379221 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.379227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.380068 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.380070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.380728 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.380738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.381198 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.381201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.381261 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.381781 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:21.381792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:21.384000 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
W20260620 07:12:21.384299 13215 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:21.378254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:52027 (local address 127.3.245.62:46131)
0620 07:12:21.378498 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:21.378505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:21.378521 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:12:21.378675 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:21.378678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:21.378780 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:21.378877 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:21.379338 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.379928 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.380850 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.381085 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.381187 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.384145 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:52027: BlockingRecv error: recv got EOF from 127.3.245.2:52027 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:21 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:21.605079 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:21.599077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:49887)
0620 07:12:21.599435 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:21.599450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:21.599551 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:21.599827 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:21.599830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:21.600019 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:21.600130 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.600136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.600921 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.600924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.601548 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:21.601558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.602046 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.602050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:21.602126 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.602724 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:21.602745 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:21.604963 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":170,"threads_started":1}
W20260620 07:12:21.605276 13215 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:21.599150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:49887 (local address 127.3.245.62:46131)
0620 07:12:21.599305 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:21.599309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:21.599518 (+ 209us) server_negotiation.cc:407] Connection header received
0620 07:12:21.599650 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:21.599652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:21.599699 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:21.599786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:21.600235 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.600817 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.601686 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:21.601912 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:21.602003 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:21.605115 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:49887: BlockingRecv error: recv got EOF from 127.3.245.3:49887 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:22.200482 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.194124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:38711)
0620 07:12:22.194464 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:22.194483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:22.194650 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:22.194967 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:22.194970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:22.195196 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:22.195288 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.195295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.196147 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.196153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.196854 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.196865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.197343 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.197347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.197432 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.198016 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:22.198029 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:22.200276 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":125,"threads_started":1}
W20260620 07:12:22.200574 13219 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:22.194215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:38711 (local address 127.3.245.62:46131)
0620 07:12:22.194531 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.194536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.194558 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:12:22.194790 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.194793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.194844 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.194945 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.195406 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.196027 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.196972 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.197227 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.197339 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.200432 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:38711: BlockingRecv error: recv got EOF from 127.3.245.1:38711 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:22.391274 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.385157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:54153)
0620 07:12:22.385448 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:22.385462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:22.385571 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:22.385953 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:22.385956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:22.386225 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:22.386326 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.386347 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.387141 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.387146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.387795 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.387806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.388340 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.388343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.388418 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.389099 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:22.389113 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:22.391140 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
W20260620 07:12:22.391527 13219 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:22.385230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:54153 (local address 127.3.245.62:46131)
0620 07:12:22.385380 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.385384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.385539 (+ 155us) server_negotiation.cc:407] Connection header received
0620 07:12:22.385765 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.385767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.385819 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.385903 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.386462 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.387008 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.387938 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.388167 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.388285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.391398 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:54153: BlockingRecv error: recv got EOF from 127.3.245.2:54153 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:22 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:22.612207 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:22.605972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:53231)
0620 07:12:22.606364 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:22.606384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:22.606534 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:22.606852 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:22.606855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:22.607068 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:22.607155 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.607161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.607913 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.607919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.608497 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:22.608505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.609013 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.609016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:22.609102 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.609632 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:22.609654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:22.612008 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":94,"threads_started":1}
W20260620 07:12:22.612313 13219 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:22.606136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:53231 (local address 127.3.245.62:46131)
0620 07:12:22.606310 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:22.606316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:22.606471 (+ 155us) server_negotiation.cc:407] Connection header received
0620 07:12:22.606636 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:22.606639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:22.606693 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:22.606837 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:22.607272 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.607800 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.608662 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:22.608893 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:22.609006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:22.612175 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:53231: BlockingRecv error: recv got EOF from 127.3.245.3:53231 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:23.207897 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:23.201438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:44701)
0620 07:12:23.201820 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:23.201835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:23.201938 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:23.202341 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:23.202344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:23.202531 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:23.202691 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.202698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.203647 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.203650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.204242 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.204251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.204855 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.204858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.204928 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.205491 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:23.205504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:23.207722 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":88,"threads_started":1}
W20260620 07:12:23.208063 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:23.201521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:44701 (local address 127.3.245.62:46131)
0620 07:12:23.201786 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:23.201792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:23.201913 (+ 121us) server_negotiation.cc:407] Connection header received
0620 07:12:23.202076 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:23.202079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:23.202145 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:23.202239 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:23.202842 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.203457 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.204380 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.204711 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.204805 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.207885 (+ 3080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:44701: BlockingRecv error: recv got EOF from 127.3.245.1:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:23.399215 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:23.392283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:35211)
0620 07:12:23.392754 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:23.392773 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:23.392896 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:23.393172 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:23.393175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:23.393360 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:23.393452 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.393457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.394455 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.394459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.395230 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.395242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.395844 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.395851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.395939 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.396529 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:23.396724 (+ 195us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:23.399061 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
W20260620 07:12:23.399425 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:23.392362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:35211 (local address 127.3.245.62:46131)
0620 07:12:23.392621 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:23.392628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:23.392886 (+ 258us) server_negotiation.cc:407] Connection header received
0620 07:12:23.392971 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:23.392974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:23.393025 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:23.393108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:23.393695 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.394328 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.395402 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.395635 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.395817 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.399298 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:35211: BlockingRecv error: recv got EOF from 127.3.245.2:35211 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:23 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:23.623694 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:23.613317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:49723)
0620 07:12:23.613733 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:23.613748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:23.613862 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:23.614216 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:23.614218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:23.614494 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:23.614593 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.614599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.615517 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.615522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.616076 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:23.616085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.616541 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.616544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:23.616657 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.621340 (+ 4683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:23.621357 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:23.623527 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
W20260620 07:12:23.623888 13223 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:23.613413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:49723 (local address 127.3.245.62:46131)
0620 07:12:23.613580 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:23.613585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:23.613831 (+ 246us) server_negotiation.cc:407] Connection header received
0620 07:12:23.613977 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:23.613980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:23.614042 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:23.614147 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:23.614756 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.615402 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.616204 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:23.616426 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:23.616538 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:23.623753 (+ 7215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:49723: BlockingRecv error: recv got EOF from 127.3.245.3:49723 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:24.214174 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:24.208763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:51085)
0620 07:12:24.209019 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:24.209033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:24.209126 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:24.209352 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:24.209354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:24.209526 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:24.209623 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.209628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.210384 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.210387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.210952 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.210960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.211447 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.211449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.211511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.212032 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:24.212044 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:24.213997 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260620 07:12:24.214298 13227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:24.208847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:51085 (local address 127.3.245.62:46131)
0620 07:12:24.209077 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:24.209081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:24.209095 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:12:24.209197 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:24.209199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:24.209242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:24.209321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:24.209733 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.210282 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.211073 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.211329 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.211429 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.214155 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:51085: BlockingRecv error: recv got EOF from 127.3.245.1:51085 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:24.406018 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:24.400223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:56435)
0620 07:12:24.400525 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:24.400539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:24.400708 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:24.400956 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:24.400958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:24.401136 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:24.401253 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.401259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.402273 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.402276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.402824 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.402831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.403259 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.403262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.403325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.403919 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:24.403931 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:24.405901 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
W20260620 07:12:24.406169 13227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:24.400366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:56435 (local address 127.3.245.62:46131)
0620 07:12:24.400537 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:24.400540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:24.400633 (+ 93us) server_negotiation.cc:407] Connection header received
0620 07:12:24.400805 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:24.400808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:24.400852 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:24.400922 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:24.401426 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.402143 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.402931 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.403140 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.403216 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.406044 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:56435: BlockingRecv error: recv got EOF from 127.3.245.2:56435 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:24 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:24.630529 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:24.624518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:60473)
0620 07:12:24.624916 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:24.624931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:24.625072 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:24.625373 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:24.625376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:24.625581 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:24.625676 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.625681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.626442 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.626447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.627044 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:24.627052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.627524 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.627527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:24.627628 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.628196 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:24.628208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:24.630354 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":116,"threads_started":1}
W20260620 07:12:24.630599 13227 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:24.624709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:60473 (local address 127.3.245.62:46131)
0620 07:12:24.624873 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:24.624877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:24.625018 (+ 141us) server_negotiation.cc:407] Connection header received
0620 07:12:24.625202 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:24.625205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:24.625263 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:24.625349 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:24.625791 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.626331 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.627172 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:24.627410 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:24.627520 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:24.630485 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:60473: BlockingRecv error: recv got EOF from 127.3.245.3:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:25.221572 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.215067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:42627)
0620 07:12:25.215378 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.215391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.215494 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.215761 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.215764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.215944 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:25.216043 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.216049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.217315 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.217320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.218007 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.218017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.218500 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.218503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.218587 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.219119 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:25.219130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:25.221362 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
W20260620 07:12:25.221648 13231 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:25.215132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:42627 (local address 127.3.245.62:46131)
0620 07:12:25.215395 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.215399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.215445 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:12:25.215578 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.215580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.215621 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.215704 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.216157 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.217194 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.218130 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.218382 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.218492 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.221501 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:42627: BlockingRecv error: recv got EOF from 127.3.245.1:42627 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":58,"threads_started":1}
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:25.413147 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.406923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:35861)
0620 07:12:25.407231 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.407245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.407342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.407617 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.407620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.407797 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:25.407891 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.407896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.408839 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.408842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.409412 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.409421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.409968 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.409971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.410041 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.410668 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:25.410681 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:25.412908 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
W20260620 07:12:25.413193 13231 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:25.407013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:35861 (local address 127.3.245.62:46131)
0620 07:12:25.407246 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.407250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.407316 (+ 66us) server_negotiation.cc:407] Connection header received
0620 07:12:25.407443 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.407446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.407491 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.407579 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.407999 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.408707 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.409593 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.409832 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.409923 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.413069 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:35861: BlockingRecv error: recv got EOF from 127.3.245.2:35861 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:25 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:25.637238 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:25.631416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:59801)
0620 07:12:25.631700 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:25.631720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:25.631824 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:25.632167 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:25.632170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:25.632386 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:25.632484 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.632491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.633405 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.633408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.633968 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:25.633975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.634498 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.634500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:25.634566 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.635074 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:25.635086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:25.637122 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
W20260620 07:12:25.637413 13231 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:25.631557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:59801 (local address 127.3.245.62:46131)
0620 07:12:25.631708 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:25.631720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:25.631774 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:12:25.631901 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:25.631903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:25.631949 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:25.632033 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:25.632698 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.633297 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.634136 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:25.634384 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:25.634474 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:25.637275 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:59801: BlockingRecv error: recv got EOF from 127.3.245.3:59801 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:26.227926 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.222473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:40275)
0620 07:12:26.222738 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.222752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:26.222833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:26.223077 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:26.223080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:26.223260 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:26.223358 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.223363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.224145 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.224148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.224772 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.224781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.225226 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.225229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.225292 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.225780 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:26.225791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:26.227785 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
W20260620 07:12:26.228097 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:26.222562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:40275 (local address 127.3.245.62:46131)
0620 07:12:26.222801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.222804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.222817 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:12:26.222899 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.222901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.222940 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.223019 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.223462 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.224030 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.224882 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.225118 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.225201 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.227936 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:40275: BlockingRecv error: recv got EOF from 127.3.245.1:40275 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:26.419589 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.414279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:51791)
0620 07:12:26.414552 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.414565 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:26.414658 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:26.414904 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:26.414906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:26.415079 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:26.415180 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.415186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.415953 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.415955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.416505 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.416513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.417032 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.417035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.417095 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.417657 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:26.417704 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:26.419474 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
W20260620 07:12:26.419817 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:26.414309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:51791 (local address 127.3.245.62:46131)
0620 07:12:26.414464 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.414467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.414636 (+ 169us) server_negotiation.cc:407] Connection header received
0620 07:12:26.414748 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.414750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.414801 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.414868 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.415286 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.415857 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.416674 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.416928 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.417020 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.419687 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:51791: BlockingRecv error: recv got EOF from 127.3.245.2:51791 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:26 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:26.644712 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:26.638207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:47387)
0620 07:12:26.638509 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:26.638526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:26.638630 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:26.638983 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:26.638985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:26.639242 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:26.639379 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.639386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.640440 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.640445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.641179 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:26.641190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.641631 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.641635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:26.641757 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.642264 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:26.642277 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:26.644510 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
W20260620 07:12:26.644937 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:26.638352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47387 (local address 127.3.245.62:46131)
0620 07:12:26.638563 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:26.638566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:26.638586 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:12:26.638760 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:26.638763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:26.638812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:26.638953 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:26.639510 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.640300 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.641299 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:26.641513 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:26.641610 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:26.644811 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:47387: BlockingRecv error: recv got EOF from 127.3.245.3:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:27.235107 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:27.228846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:43359)
0620 07:12:27.229298 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:27.229312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:27.229408 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:27.229736 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:27.229738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:27.229930 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:27.230063 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.230070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.230885 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.230888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.231576 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.231588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.232124 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.232127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.232213 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.232799 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:27.232813 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:27.234885 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":104,"threads_started":1}
W20260620 07:12:27.235157 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:27.229075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:43359 (local address 127.3.245.62:46131)
0620 07:12:27.229366 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:27.229370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:27.229386 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:12:27.229540 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:27.229544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:27.229603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:27.229693 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:27.230178 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.230751 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.231710 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.231965 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.232066 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.235021 (+ 2955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:43359: BlockingRecv error: recv got EOF from 127.3.245.1:43359 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:27.425972 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:27.420424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:35207)
0620 07:12:27.420779 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:27.420792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:27.420887 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:27.421128 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:27.421131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:27.421308 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:27.421399 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.421405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.422172 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.422174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.422715 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.422722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.423180 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.423182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.423240 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.423778 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:27.423791 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:27.425836 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":86,"threads_started":1}
W20260620 07:12:27.426100 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:27.420501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:35207 (local address 127.3.245.62:46131)
0620 07:12:27.420726 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:27.420730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:27.420861 (+ 131us) server_negotiation.cc:407] Connection header received
0620 07:12:27.420964 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:27.420967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:27.421015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:27.421087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:27.421552 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.422068 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.422838 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.423077 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.423166 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.425982 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:35207: BlockingRecv error: recv got EOF from 127.3.245.2:35207 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:27 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:27.651877 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:27.645748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:51657)
0620 07:12:27.646115 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:27.646131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:27.646283 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:27.646584 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:27.646587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:27.646871 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:27.646966 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.646972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.647710 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.647713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.648271 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:27.648278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.648781 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.648784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:27.648848 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.649368 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:27.649380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:27.651699 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":95,"threads_started":1}
W20260620 07:12:27.652069 13239 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:27.645836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:51657 (local address 127.3.245.62:46131)
0620 07:12:27.645991 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:27.645996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:27.646253 (+ 257us) server_negotiation.cc:407] Connection header received
0620 07:12:27.646391 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:27.646394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:27.646456 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:27.646545 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:27.647077 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.647598 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.648380 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:27.648682 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:27.648765 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:27.651894 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:51657: BlockingRecv error: recv got EOF from 127.3.245.3:51657 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:28.243180 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:28.235973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:36437)
0620 07:12:28.236371 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:28.236386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:28.236562 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:28.237005 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:28.237010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:28.237273 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:28.237425 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.237436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.238431 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.238437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.239491 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.239507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.240161 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.240167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.240283 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.240865 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:28.240887 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:28.242999 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":110,"threads_started":1}
W20260620 07:12:28.243255 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:28.236011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:36437 (local address 127.3.245.62:46131)
0620 07:12:28.236292 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:28.236295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:28.236468 (+ 173us) server_negotiation.cc:407] Connection header received
0620 07:12:28.236709 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:28.236712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:28.236793 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:28.236911 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:28.237567 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.238317 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.239628 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.240031 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.240149 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.243125 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:36437: BlockingRecv error: recv got EOF from 127.3.245.1:36437 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":124,"threads_started":1}
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:28.432678 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:28.426956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:49773)
0620 07:12:28.427332 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:28.427344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:28.427532 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:28.427811 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:28.427814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:28.428026 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:28.428113 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.428119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.428942 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.428948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.429526 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.429533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.429977 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.429981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.430056 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.430589 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:28.430600 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:28.432468 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":104,"threads_started":1}
W20260620 07:12:28.432725 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:28.427033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:49773 (local address 127.3.245.62:46131)
0620 07:12:28.427193 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:28.427200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:28.427425 (+ 225us) server_negotiation.cc:407] Connection header received
0620 07:12:28.427645 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:28.427649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:28.427702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:28.427794 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:28.428231 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.428838 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.429646 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.429867 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.429982 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.432597 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:49773: BlockingRecv error: recv got EOF from 127.3.245.2:49773 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:28 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:28.661844 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:28.652788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:52489)
0620 07:12:28.653139 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:28.653153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:28.653257 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:28.653582 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:28.653585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:28.653772 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:28.653880 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.653886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.654637 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.654639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.655212 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:28.655221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.655702 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.655704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:28.655769 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.659461 (+ 3692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:28.659476 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:28.661723 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
W20260620 07:12:28.662101 13242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:28.652845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:52489 (local address 127.3.245.62:46131)
0620 07:12:28.653003 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:28.653008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:28.653239 (+ 231us) server_negotiation.cc:407] Connection header received
0620 07:12:28.653327 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:28.653329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:28.653383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:28.653454 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:28.654014 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.654536 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.655318 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:28.655525 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:28.655706 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:28.661904 (+ 6198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:52489: BlockingRecv error: recv got EOF from 127.3.245.3:52489 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:29.250125 13246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.244349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:41451)
0620 07:12:29.244738 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.244752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.244885 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.245250 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.245254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.245497 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.245607 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.245613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.246368 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.246371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.246968 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.246999 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.247443 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.247446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.247512 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.248004 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.248017 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.249950 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":94,"threads_started":1}
W20260620 07:12:29.250211 13247 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:29.244473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:41451 (local address 127.3.245.62:46131)
0620 07:12:29.244735 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.244740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.244830 (+ 90us) server_negotiation.cc:407] Connection header received
0620 07:12:29.245027 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.245030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.245090 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.245181 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.245715 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.246257 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.247105 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.247345 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.247428 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.250081 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:41451: BlockingRecv error: recv got EOF from 127.3.245.1:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":57,"threads_started":1}
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:29.439827 13248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.433712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:56283)
0620 07:12:29.434012 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.434026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.434133 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.434408 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.434411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.434598 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.434694 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.434699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.435699 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.435702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.436509 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.436521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.437062 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.437065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.437137 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.437667 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.437680 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.439642 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
W20260620 07:12:29.439872 13247 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:29.433868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:56283 (local address 127.3.245.62:46131)
0620 07:12:29.434014 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.434018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.434121 (+ 103us) server_negotiation.cc:407] Connection header received
0620 07:12:29.434213 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.434215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.434262 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.434341 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.434844 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.435537 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.436685 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.436925 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.437007 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.439764 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:56283: BlockingRecv error: recv got EOF from 127.3.245.2:56283 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:29 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:29.668823 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:29.662702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:47343)
0620 07:12:29.662996 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:29.663011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:29.663129 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:29.663471 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:29.663474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:29.663666 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:29.663773 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.663780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.664606 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.664609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.665332 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:29.665342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.665824 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.665826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:29.665892 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.666489 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:29.666502 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:29.668590 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
W20260620 07:12:29.668974 13247 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:29.662766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47343 (local address 127.3.245.62:46131)
0620 07:12:29.663020 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:29.663024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:29.663071 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:12:29.663252 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:29.663255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:29.663307 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:29.663389 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:29.663891 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.664481 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.665462 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:29.665712 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:29.665802 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:29.668805 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:47343: BlockingRecv error: recv got EOF from 127.3.245.3:47343 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:30.257490 13250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.251121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:54851)
0620 07:12:30.251409 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.251422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.251520 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.251809 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.251811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.252002 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.252100 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.252106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.252998 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.253001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.253695 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.253706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.254365 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.254370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.254474 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.255085 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.255098 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.257362 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
W20260620 07:12:30.257736 13251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:30.251238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:54851 (local address 127.3.245.62:46131)
0620 07:12:30.251577 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.251581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.251596 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:12:30.251638 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.251640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.251682 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.251774 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.252230 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.252877 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.253828 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.254126 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.254305 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.257584 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:54851: BlockingRecv error: recv got EOF from 127.3.245.1:54851 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:30.447136 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.440757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:42231)
0620 07:12:30.441125 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.441144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.441326 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.441638 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.441641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.441879 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.441983 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.441989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.442879 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.442886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.443525 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.443534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.444050 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.444054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.444148 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.444778 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.444790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.446946 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
W20260620 07:12:30.447172 13251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:30.440968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:42231 (local address 127.3.245.62:46131)
0620 07:12:30.441131 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.441139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.441246 (+ 107us) server_negotiation.cc:407] Connection header received
0620 07:12:30.441442 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.441455 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.441513 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.441614 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.442116 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.442753 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.443670 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.443915 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.444033 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.447067 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:42231: BlockingRecv error: recv got EOF from 127.3.245.2:42231 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:30 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.3@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:30.676141 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:30.669733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.3:47727)
0620 07:12:30.670078 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:30.670094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:30.670261 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:30.670547 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:30.670550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:30.670761 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:30.670847 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.670852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.671663 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.671668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.672318 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:30.672328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.673016 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.673023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:30.673104 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.673732 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:30.673753 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:30.675991 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
W20260620 07:12:30.676214 13251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:30.669913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47727 (local address 127.3.245.62:46131)
0620 07:12:30.670078 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:30.670084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:30.670196 (+ 112us) server_negotiation.cc:407] Connection header received
0620 07:12:30.670380 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:30.670387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:30.670432 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:30.670532 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:30.670969 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.671540 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.672489 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:30.672796 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:30.672946 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:30.676116 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.3:47727: BlockingRecv error: recv got EOF from 127.3.245.3:47727 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 07:12:31 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:31 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.1@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:31.265453 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:31.258623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.1:51115)
0620 07:12:31.258967 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:31.258983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:31.259076 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:31.259428 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:31.259431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:31.259677 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:31.259793 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:31.259804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.260877 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.260881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:31.261687 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:31.261698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.262396 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.262399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:31.262470 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:31.263166 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:31.263181 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:31.265324 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
W20260620 07:12:31.265638 13255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:31.258700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:51115 (local address 127.3.245.62:46131)
0620 07:12:31.259034 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:31.259040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:31.259067 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:12:31.259223 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:31.259226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:31.259273 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:31.259420 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:31.259924 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.260748 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.261796 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.262104 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.262206 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:31.265493 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.1:51115: BlockingRecv error: recv got EOF from 127.3.245.1:51115 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
Jun 20 07:12:31 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 07:12:31 dist-test-slave-ldfr krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.245.2@KRBTEST.COM for oryx/127.3.245.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 07:12:31.454243 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:31.448053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:46131 (local address 127.3.245.2:46441)
0620 07:12:31.448415 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:31.448437 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:31.448562 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:31.448898 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:31.448902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:31.449127 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:31.449211 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:31.449217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.450107 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.450112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:31.451017 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:31.451034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.451482 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.451486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:31.451548 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:31.452118 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:31.452132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:31.454097 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.245.62:46131: Server oryx/127.3.245.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
W20260620 07:12:31.454367 13255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:31.448187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:46441 (local address 127.3.245.62:46131)
0620 07:12:31.448372 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:31.448376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:31.448516 (+ 140us) server_negotiation.cc:407] Connection header received
0620 07:12:31.448725 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:31.448740 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:31.448785 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:31.448902 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:31.449326 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.449960 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.451138 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.451358 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.451458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:31.454231 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.245.2:46441: BlockingRecv error: recv got EOF from 127.3.245.2:46441 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:12:31.557024 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12780
I20260620 07:12:31.562970 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12932
I20260620 07:12:31.568449 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13065
I20260620 07:12:31.574214 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 12710
2026-06-20T07:12:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18056 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T07:12:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:31Z Disabled control of system clock
I20260620 07:12:31.606336 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36431
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 07:12:31.718459 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:31.718724 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:31.718760 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:31.722699 13260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:31.722756 13260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:31.722772 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:31.722788 13260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:31.722803 13260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:31.727600 13260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:31.728641 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:31.729967 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:31.738857 13265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:31.738868 13266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:31.738852 13268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:31.739012 13260 server_base.cc:1034] running on GCE node
I20260620 07:12:31.739636 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:31.740850 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:31.742012 13260 hybrid_clock.cc:648] HybridClock initialized: now 1781939551741994 us; error 32 us; skew 500 ppm
I20260620 07:12:31.743775 13260 webserver.cc:460] Webserver started at http://127.3.245.62:44653/ using document root <none> and password file <none>
I20260620 07:12:31.744372 13260 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:31.744434 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:31.744752 13260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:31.746815 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "0e9604b9f7f9427190cd905ddc75a8ff"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "2d9d65d9f68cedf804ba0de9554f4d2b"
server_key_iv: "31f9a1eecc97b065b466cf0a03e1d4c0"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.747396 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "0e9604b9f7f9427190cd905ddc75a8ff"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "2d9d65d9f68cedf804ba0de9554f4d2b"
server_key_iv: "31f9a1eecc97b065b466cf0a03e1d4c0"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.751071 13260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:12:31.753517 13274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:31.754704 13260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 07:12:31.754837 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "0e9604b9f7f9427190cd905ddc75a8ff"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "2d9d65d9f68cedf804ba0de9554f4d2b"
server_key_iv: "31f9a1eecc97b065b466cf0a03e1d4c0"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.754952 13260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:31.767482 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:31.768213 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:31.768414 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:31.775769 13260 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:36431
I20260620 07:12:31.775840 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:36431 every 8 connection(s)
I20260620 07:12:31.776876 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:31.779822 13327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:31.782477 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13260
I20260620 07:12:31.782577 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:31.782857 4052 external_mini_cluster.cc:1428] Setting key 07b74ff3dca6c7d22e9027c37f656701
I20260620 07:12:31.785887 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff: Bootstrap starting.
I20260620 07:12:31.788511 13327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:31.789379 13327 log.cc:824] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:31.791553 13327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff: No bootstrap required, opened a new log
I20260620 07:12:31.792428 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:31.784295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.3.245.62:36431)
0620 07:12:31.784920 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:31.784933 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:31.784967 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:12:31.785563 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:31.785579 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:31.785961 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:31.786295 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:31.786692 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.787649 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.788921 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:31.789376 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:31.789531 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:31.790415 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:31.790421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:31.790437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:31.790906 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:31.790981 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:31.791007 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:31.791116 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:31.791255 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 07:12:31.791510 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
I20260620 07:12:31.795146 13327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } }
I20260620 07:12:31.795356 13327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:31.795388 13327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9604b9f7f9427190cd905ddc75a8ff, State: Initialized, Role: FOLLOWER
I20260620 07:12:31.795804 13327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } }
I20260620 07:12:31.795922 13327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:31.795962 13327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:31.796031 13327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:31.796954 13327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } }
I20260620 07:12:31.797243 13327 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9604b9f7f9427190cd905ddc75a8ff; no voters:
I20260620 07:12:31.797501 13327 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:31.797637 13332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:31.797890 13332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [term 1 LEADER]: Becoming Leader. State: Replica: 0e9604b9f7f9427190cd905ddc75a8ff, State: Running, Role: LEADER
I20260620 07:12:31.798242 13332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } }
I20260620 07:12:31.798679 13327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:31.800031 13334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9604b9f7f9427190cd905ddc75a8ff. Latest consensus state: current_term: 1 leader_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } } }
I20260620 07:12:31.800163 13334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:31.800578 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9604b9f7f9427190cd905ddc75a8ff" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36431 } } }
I20260620 07:12:31.800707 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:31.801187 13341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:31.803725 13341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:31.808900 13341 catalog_manager.cc:1347] Generated new cluster ID: f6be82c36b974423b667e25d393292e0
I20260620 07:12:31.808971 13341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:31.831413 13341 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:31.832834 13341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:31.839332 13341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9604b9f7f9427190cd905ddc75a8ff: Generated new TSK 0
I20260620 07:12:31.840126 13341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:31.843250 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--builtin_ntp_servers=127.3.245.20:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 07:12:31.956501 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:31.956854 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:31.956929 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:31.961236 13351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:31.961337 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:31.961431 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:31.966400 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:31.967569 13351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:31.969036 13351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:31.978749 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:31.978736 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:31.978736 13351 server_base.cc:1034] running on GCE node
W20260620 07:12:31.978775 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:31.979537 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:31.980129 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:31.981329 13351 hybrid_clock.cc:648] HybridClock initialized: now 1781939551981306 us; error 35 us; skew 500 ppm
I20260620 07:12:31.983287 13351 webserver.cc:460] Webserver started at http://127.3.245.1:43399/ using document root <none> and password file <none>
I20260620 07:12:31.983863 13351 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:31.983927 13351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:31.984157 13351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:31.986315 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "2a137248d9884a289dc8e4b1036e8094"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "4c9580ef4cd6a691af756ed810336109"
server_key_iv: "fc653cec4ca68057367e229a7adb44f7"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.986876 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "2a137248d9884a289dc8e4b1036e8094"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "4c9580ef4cd6a691af756ed810336109"
server_key_iv: "fc653cec4ca68057367e229a7adb44f7"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.990586 13351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:12:31.993352 13365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:31.994774 13351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:31.994935 13351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "2a137248d9884a289dc8e4b1036e8094"
format_stamp: "Formatted at 2026-06-20 07:12:31 on dist-test-slave-ldfr"
server_key: "4c9580ef4cd6a691af756ed810336109"
server_key_iv: "fc653cec4ca68057367e229a7adb44f7"
server_key_version: "encryptionkey@0"
I20260620 07:12:31.995065 13351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:32.016712 13351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:32.017530 13351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:32.017760 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:32.018509 13351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:32.019685 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:32.019765 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.019845 13351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:32.019901 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.029920 13351 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:44279
I20260620 07:12:32.029989 13478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:44279 every 8 connection(s)
I20260620 07:12:32.030992 13351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:32.040344 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.033043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:39117 (local address 127.3.245.62:36431)
0620 07:12:32.033190 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.033194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.034268 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 07:12:32.035690 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.035693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.035738 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.035813 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.036793 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.037335 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.038491 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.038713 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.038797 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.039921 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:32.039924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:32.039926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:32.039938 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:32.039964 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:32.039978 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:32.040041 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:32.040185 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 07:12:32.040234 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:12:32.040534 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13351
I20260620 07:12:32.040688 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:32.041034 4052 external_mini_cluster.cc:1428] Setting key 66bfaac566fc8cbb855f44f23a194b23
I20260620 07:12:32.041692 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.033423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36431 (local address 127.3.245.1:39117)
0620 07:12:32.034110 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:32.034151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:32.035497 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:32.035975 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:32.035982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:32.036064 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:32.036592 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.036605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.037492 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.037495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.038366 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.038375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.038875 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.038878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.039061 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.039662 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:32.039663 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:32.039705 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:32.039770 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:32.040058 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:32.040064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:32.040069 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:12:32.040173 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:12:32.040529 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260620 07:12:32.043027 13479 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36431
I20260620 07:12:32.043241 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--builtin_ntp_servers=127.3.245.20:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:12:32.043407 13479 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:32.044040 13479 heartbeater.cc:511] Master 127.3.245.62:36431 requested a full tablet report, sending...
I20260620 07:12:32.045866 13290 ts_manager.cc:194] Registered new tserver with Master: 2a137248d9884a289dc8e4b1036e8094 (127.3.245.1:44279)
I20260620 07:12:32.047605 13290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.1:39117
W20260620 07:12:32.156083 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:32.156360 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:32.156399 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:32.160597 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:32.160677 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:32.160734 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:32.165587 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:32.166646 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:32.167999 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:32.177639 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.177601 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.177603 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:32.178054 13482 server_base.cc:1034] running on GCE node
I20260620 07:12:32.178458 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:32.179121 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:32.180330 13482 hybrid_clock.cc:648] HybridClock initialized: now 1781939552180300 us; error 48 us; skew 500 ppm
I20260620 07:12:32.182348 13482 webserver.cc:460] Webserver started at http://127.3.245.2:33499/ using document root <none> and password file <none>
I20260620 07:12:32.182932 13482 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:32.182996 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:32.183214 13482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:32.185374 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "7092b9f843df450894d3545c714d2175"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "a467075ec18ace833d760aa3e6adaad2"
server_key_iv: "f42d246773c8f9893d3192808f6435d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.185926 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "7092b9f843df450894d3545c714d2175"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "a467075ec18ace833d760aa3e6adaad2"
server_key_iv: "f42d246773c8f9893d3192808f6435d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.189621 13482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:12:32.192227 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.193514 13482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:12:32.193666 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "7092b9f843df450894d3545c714d2175"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "a467075ec18ace833d760aa3e6adaad2"
server_key_iv: "f42d246773c8f9893d3192808f6435d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.193778 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:32.210156 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:32.210965 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:32.211184 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:32.211796 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:32.212818 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:32.212891 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.212963 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:32.213009 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.222838 13482 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44441
I20260620 07:12:32.222883 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44441 every 8 connection(s)
I20260620 07:12:32.223780 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:32.230794 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13482
I20260620 07:12:32.230929 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:32.231205 4052 external_mini_cluster.cc:1428] Setting key 8e4d2d74eba0e4a9175c2089cc8780f8
I20260620 07:12:32.233304 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--builtin_ntp_servers=127.3.245.20:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 07:12:32.234285 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.225744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:40083 (local address 127.3.245.62:36431)
0620 07:12:32.225905 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.225910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.227197 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 07:12:32.228831 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.228835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.228906 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.229037 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.230082 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.230683 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.232137 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.232368 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.232481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.233773 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:32.233782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:32.233784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:32.233795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:32.233813 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:32.233820 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:32.233914 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:32.234076 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 07:12:32.234149 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:12:32.235209 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.226221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36431 (local address 127.3.245.2:40083)
0620 07:12:32.226991 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:32.227047 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:32.228556 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:32.229260 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:32.229268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:32.229366 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:32.229906 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.229920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.231010 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.231017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.232007 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.232017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.232519 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.232523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.232764 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.233493 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:32.233495 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:32.233557 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:32.233622 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:32.233918 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:32.233929 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:32.233937 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:12:32.234075 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 07:12:32.234396 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260620 07:12:32.236351 13610 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36431
I20260620 07:12:32.236591 13610 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:32.237135 13610 heartbeater.cc:511] Master 127.3.245.62:36431 requested a full tablet report, sending...
I20260620 07:12:32.238216 13290 ts_manager.cc:194] Registered new tserver with Master: 7092b9f843df450894d3545c714d2175 (127.3.245.2:44441)
I20260620 07:12:32.239081 13290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.2:40083
W20260620 07:12:32.351186 13613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:32.351457 13613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:32.351495 13613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:32.355528 13613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:32.355594 13613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:32.355650 13613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:32.360520 13613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:32.361608 13613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:32.362929 13613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:32.372889 13619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.372917 13621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.372889 13618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:32.373142 13613 server_base.cc:1034] running on GCE node
I20260620 07:12:32.373684 13613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:32.374439 13613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:32.375646 13613 hybrid_clock.cc:648] HybridClock initialized: now 1781939552375633 us; error 27 us; skew 500 ppm
I20260620 07:12:32.378029 13613 webserver.cc:460] Webserver started at http://127.3.245.3:43795/ using document root <none> and password file <none>
I20260620 07:12:32.378693 13613 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:32.378799 13613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:32.379066 13613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:32.381440 13613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "183c807dc51e433d9432d9b8082fe4d4"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "4c38a5472a57bb835a8515ca02f05ca7"
server_key_iv: "dd0e6878320e9b3bab0ccaf256a2db41"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.382057 13613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "183c807dc51e433d9432d9b8082fe4d4"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "4c38a5472a57bb835a8515ca02f05ca7"
server_key_iv: "dd0e6878320e9b3bab0ccaf256a2db41"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.386214 13613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:12:32.389278 13627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.390877 13613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:12:32.391072 13613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "183c807dc51e433d9432d9b8082fe4d4"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "4c38a5472a57bb835a8515ca02f05ca7"
server_key_iv: "dd0e6878320e9b3bab0ccaf256a2db41"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.391234 13613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:32.404762 13613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:32.405508 13613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:32.405726 13613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:32.406373 13613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:32.407405 13613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:32.407481 13613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.407563 13613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:32.407605 13613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.417613 13613 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:44765
I20260620 07:12:32.417573 13740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:44765 every 8 connection(s)
I20260620 07:12:32.418567 13613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:32.420267 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13613
I20260620 07:12:32.420393 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:32.420782 4052 external_mini_cluster.cc:1428] Setting key 66128f6d007d91a970af3fe028da768d
I20260620 07:12:32.429486 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.420778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:58933 (local address 127.3.245.62:36431)
0620 07:12:32.420939 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.420943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.422614 (+ 1671us) server_negotiation.cc:407] Connection header received
0620 07:12:32.424224 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.424226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.424276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.424354 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.425431 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.426035 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.427308 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.427571 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.427671 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.428998 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:32.429002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:32.429004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:32.429019 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:32.429044 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:32.429053 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:32.429130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:32.429290 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:12:32.429353 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:12:32.430413 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.421405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36431 (local address 127.3.245.3:58933)
0620 07:12:32.422350 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:32.422490 (+ 140us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:32.424002 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:32.424583 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:32.424591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:32.424761 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:32.425274 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.425288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.426179 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.426183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.427179 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:32.427190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:32.427739 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:32.427742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:32.427901 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:32.428689 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:32.428691 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:32.428753 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:32.428827 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:32.429139 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:32.429145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:32.429149 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:12:32.429244 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 07:12:32.429488 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":160,"threads_started":1}
I20260620 07:12:32.431843 13741 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36431
I20260620 07:12:32.432117 13741 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:32.432770 13741 heartbeater.cc:511] Master 127.3.245.62:36431 requested a full tablet report, sending...
I20260620 07:12:32.434015 13290 ts_manager.cc:194] Registered new tserver with Master: 183c807dc51e433d9432d9b8082fe4d4 (127.3.245.3:44765)
I20260620 07:12:32.434777 13290 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.3:58933
I20260620 07:12:32.435441 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:12:32.438424 13330 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:32.437507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.3.245.62:36431)
0620 07:12:32.437644 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.437656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.437806 (+ 150us) server_negotiation.cc:407] Connection header received
0620 07:12:32.437911 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.437913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.437954 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.438014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.438289 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52980: BlockingRecv error: recv got EOF from 127.0.0.1:52980 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:12:32.438597 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13351
I20260620 07:12:32.444427 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13482
I20260620 07:12:32.449947 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13613
I20260620 07:12:32.455230 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13260
2026-06-20T07:12:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T07:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:32Z Disabled control of system clock
I20260620 07:12:32.486119 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34691
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:12:32.600750 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:32.601073 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:32.601112 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:32.605304 13752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:32.605384 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:32.605400 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:32.605418 13752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:32.605434 13752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:32.610677 13752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:32.611850 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:32.613379 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:32.622571 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.622637 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.622591 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:32.622814 13752 server_base.cc:1034] running on GCE node
I20260620 07:12:32.623589 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:32.624977 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:32.626181 13752 hybrid_clock.cc:648] HybridClock initialized: now 1781939552626164 us; error 36 us; skew 500 ppm
I20260620 07:12:32.628479 13752 webserver.cc:460] Webserver started at http://127.3.245.62:38989/ using document root <none> and password file <none>
I20260620 07:12:32.629298 13752 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:32.629372 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:32.629671 13752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:32.632597 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "ee6e7fd0f5cf4203a0ec64540c49d911"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "02cac3eb5d9094b277fab00a5f75024c"
server_key_iv: "a88e1cc409f77223494280944d7353b9"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.633262 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "ee6e7fd0f5cf4203a0ec64540c49d911"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "02cac3eb5d9094b277fab00a5f75024c"
server_key_iv: "a88e1cc409f77223494280944d7353b9"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.637154 13752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:12:32.639701 13766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.641070 13752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:12:32.641207 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "ee6e7fd0f5cf4203a0ec64540c49d911"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "02cac3eb5d9094b277fab00a5f75024c"
server_key_iv: "a88e1cc409f77223494280944d7353b9"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.641331 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:32.646128 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:32.646770 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:32.646957 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:32.654085 13752 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:34691
I20260620 07:12:32.654094 13818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:34691 every 8 connection(s)
I20260620 07:12:32.655074 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:32.658502 13819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:32.662585 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13752
I20260620 07:12:32.662683 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:32.663029 4052 external_mini_cluster.cc:1428] Setting key 28e0e9c177babe985dd09a20755f2866
I20260620 07:12:32.664565 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911: Bootstrap starting.
I20260620 07:12:32.667500 13819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:32.668454 13819 log.cc:824] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:32.669775 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.664600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.3.245.62:34691)
0620 07:12:32.665439 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.665464 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.665507 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:12:32.666415 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.666439 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.666887 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.667263 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.668189 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:32.668195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:32.668212 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:32.668227 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:32.668298 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:32.668322 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:32.668439 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:32.668693 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 07:12:32.668968 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":140,"threads_started":1}
I20260620 07:12:32.670675 13819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911: No bootstrap required, opened a new log
I20260620 07:12:32.674345 13819 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } }
I20260620 07:12:32.674571 13819 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:32.674631 13819 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6e7fd0f5cf4203a0ec64540c49d911, State: Initialized, Role: FOLLOWER
I20260620 07:12:32.675043 13819 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } }
I20260620 07:12:32.675163 13819 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:32.675235 13819 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:32.675334 13819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:32.676409 13819 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } }
I20260620 07:12:32.676796 13819 leader_election.cc:302] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee6e7fd0f5cf4203a0ec64540c49d911; no voters:
I20260620 07:12:32.677109 13819 leader_election.cc:288] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:32.677260 13824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:32.677611 13824 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [term 1 LEADER]: Becoming Leader. State: Replica: ee6e7fd0f5cf4203a0ec64540c49d911, State: Running, Role: LEADER
I20260620 07:12:32.677942 13824 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } }
I20260620 07:12:32.678421 13819 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:32.679733 13826 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee6e7fd0f5cf4203a0ec64540c49d911. Latest consensus state: current_term: 1 leader_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } } }
I20260620 07:12:32.679860 13826 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:32.680167 13825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6e7fd0f5cf4203a0ec64540c49d911" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34691 } } }
I20260620 07:12:32.680269 13825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911 [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:32.680522 13833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:32.683511 13833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:32.689409 13833 catalog_manager.cc:1347] Generated new cluster ID: 3e7abd86fc154bdc94400b710f3b4461
I20260620 07:12:32.689491 13833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:32.699854 13833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:32.700209 13833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:32.708223 13833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee6e7fd0f5cf4203a0ec64540c49d911: Generated new TSK 0
I20260620 07:12:32.709005 13833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:12:32.715014 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--builtin_ntp_servers=127.3.245.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:12:32.828701 13843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:32.828969 13843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:32.829011 13843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:32.833335 13843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:32.833398 13843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:32.833458 13843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:32.838454 13843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:32.839596 13843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:32.841089 13843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:32.851056 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:32.851178 13843 server_base.cc:1034] running on GCE node
W20260620 07:12:32.851056 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:32.851367 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:32.851838 13843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:32.852535 13843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:32.853732 13843 hybrid_clock.cc:648] HybridClock initialized: now 1781939552853697 us; error 49 us; skew 500 ppm
I20260620 07:12:32.855659 13843 webserver.cc:460] Webserver started at http://127.3.245.1:45105/ using document root <none> and password file <none>
I20260620 07:12:32.856290 13843 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:32.856391 13843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:32.856714 13843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:32.859009 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "eeea5bf691bc4f4f90af6db63b37ffe3"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "12dadfe93f85949144b4ef5b2ce5e7c4"
server_key_iv: "2198f781985714c7e8eee81862337844"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.859650 13843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "eeea5bf691bc4f4f90af6db63b37ffe3"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "12dadfe93f85949144b4ef5b2ce5e7c4"
server_key_iv: "2198f781985714c7e8eee81862337844"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.863708 13843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:12:32.866650 13857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.867919 13843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:32.868073 13843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "eeea5bf691bc4f4f90af6db63b37ffe3"
format_stamp: "Formatted at 2026-06-20 07:12:32 on dist-test-slave-ldfr"
server_key: "12dadfe93f85949144b4ef5b2ce5e7c4"
server_key_iv: "2198f781985714c7e8eee81862337844"
server_key_version: "encryptionkey@0"
I20260620 07:12:32.868208 13843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:32.873330 13843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:32.874011 13843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:32.874217 13843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:32.874804 13843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:32.875792 13843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:32.875869 13843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.875945 13843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:32.875994 13843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:32.886116 13843 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:35769
I20260620 07:12:32.886134 13970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:35769 every 8 connection(s)
I20260620 07:12:32.887099 13843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:32.891867 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13843
I20260620 07:12:32.892014 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:32.892323 4052 external_mini_cluster.cc:1428] Setting key 38f0f5c315afbebb6e9ec57106cfcdee
I20260620 07:12:32.894060 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.889127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:33147 (local address 127.3.245.62:34691)
0620 07:12:32.889279 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:32.889283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:32.890409 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 07:12:32.891828 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:32.891832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:32.891881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:32.892010 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:32.893462 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:32.893465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:32.893467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:32.893478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:32.893498 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:32.893506 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:32.893584 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:32.893789 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 07:12:32.893867 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:12:32.894750 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--builtin_ntp_servers=127.3.245.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 07:12:32.895049 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:32.889547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34691 (local address 127.3.245.1:33147)
0620 07:12:32.890250 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:32.890293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:32.891566 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:32.892262 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:32.892272 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:32.892388 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:32.893148 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:32.893150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:32.893192 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:32.893302 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:32.893601 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:32.893610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:32.893616 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:12:32.893793 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 07:12:32.894085 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260620 07:12:32.896507 13971 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34691
I20260620 07:12:32.896813 13971 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:32.897332 13971 heartbeater.cc:511] Master 127.3.245.62:34691 requested a full tablet report, sending...
I20260620 07:12:32.899327 13783 ts_manager.cc:194] Registered new tserver with Master: eeea5bf691bc4f4f90af6db63b37ffe3 (127.3.245.1:35769)
W20260620 07:12:33.007294 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:33.007565 13974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:33.007606 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:33.011831 13974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:33.011900 13974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:33.012002 13974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:33.017053 13974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:33.018193 13974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:33.019598 13974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:33.029510 13982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:33.029526 13979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:33.029507 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:33.030112 13974 server_base.cc:1034] running on GCE node
I20260620 07:12:33.030546 13974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:33.031315 13974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:33.032531 13974 hybrid_clock.cc:648] HybridClock initialized: now 1781939553032521 us; error 33 us; skew 500 ppm
I20260620 07:12:33.034566 13974 webserver.cc:460] Webserver started at http://127.3.245.2:36467/ using document root <none> and password file <none>
I20260620 07:12:33.035175 13974 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:33.035275 13974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:33.035506 13974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:33.037714 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "725add5d14644be1b284769984ddb4b9"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3b7ea1f9e018cb15b42abb20df33587d"
server_key_iv: "c28571480ee3b0cda9e79ed11222cc37"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.038283 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "725add5d14644be1b284769984ddb4b9"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3b7ea1f9e018cb15b42abb20df33587d"
server_key_iv: "c28571480ee3b0cda9e79ed11222cc37"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.042011 13974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:12:33.044570 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.046015 13974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:12:33.046164 13974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "725add5d14644be1b284769984ddb4b9"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3b7ea1f9e018cb15b42abb20df33587d"
server_key_iv: "c28571480ee3b0cda9e79ed11222cc37"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.046280 13974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:33.050937 13974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:33.051545 13974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:33.051731 13974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:33.052304 13974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:33.053268 13974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:33.053339 13974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.053419 13974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:33.053470 13974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.062049 13974 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:34767
I20260620 07:12:33.062090 14101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:34767 every 8 connection(s)
I20260620 07:12:33.062987 13974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:33.069397 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:33.064899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:44645 (local address 127.3.245.62:34691)
0620 07:12:33.065055 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:33.065060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:33.066300 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 07:12:33.067628 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:33.067631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:33.067668 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:33.067750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:33.068964 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:33.068967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:33.068969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:33.068980 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:33.068999 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:33.069007 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:33.069094 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:33.069244 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 07:12:33.069286 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:12:33.070230 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:33.065380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34691 (local address 127.3.245.2:44645)
0620 07:12:33.066122 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:33.066164 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:33.067411 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:33.067974 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:33.067981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:33.068068 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:33.068657 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:33.068661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:33.068715 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:33.068785 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:33.069090 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:33.069097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:33.069101 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:12:33.069233 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:12:33.069420 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260620 07:12:33.071350 14102 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34691
I20260620 07:12:33.071624 14102 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:33.072046 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 13974
I20260620 07:12:33.072078 14102 heartbeater.cc:511] Master 127.3.245.62:34691 requested a full tablet report, sending...
I20260620 07:12:33.072263 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:33.072576 4052 external_mini_cluster.cc:1428] Setting key 11548bd3ca32e13f9e00910af5197257
I20260620 07:12:33.073285 13783 ts_manager.cc:194] Registered new tserver with Master: 725add5d14644be1b284769984ddb4b9 (127.3.245.2:34767)
I20260620 07:12:33.074647 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--builtin_ntp_servers=127.3.245.20:42381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 07:12:33.189800 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:33.190048 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:33.190088 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:33.194039 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:33.194092 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:33.194149 14105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:33.198778 14105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:33.199819 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:33.201205 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:33.210497 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:33.210502 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:33.210621 14105 server_base.cc:1034] running on GCE node
W20260620 07:12:33.210568 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:33.211153 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:33.211830 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:33.213054 14105 hybrid_clock.cc:648] HybridClock initialized: now 1781939553213036 us; error 28 us; skew 500 ppm
I20260620 07:12:33.215056 14105 webserver.cc:460] Webserver started at http://127.3.245.3:45823/ using document root <none> and password file <none>
I20260620 07:12:33.215675 14105 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:33.215767 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:33.216001 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:33.218101 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "f74cc9aa8eaa4407b7d839fd4712555f"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3e3cfe5da191c114cd921d41eae28454"
server_key_iv: "e73504993c1254c015c78e8575e09bb9"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.218667 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "f74cc9aa8eaa4407b7d839fd4712555f"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3e3cfe5da191c114cd921d41eae28454"
server_key_iv: "e73504993c1254c015c78e8575e09bb9"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.222543 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:12:33.225458 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.226766 14105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:12:33.226893 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "f74cc9aa8eaa4407b7d839fd4712555f"
format_stamp: "Formatted at 2026-06-20 07:12:33 on dist-test-slave-ldfr"
server_key: "3e3cfe5da191c114cd921d41eae28454"
server_key_iv: "e73504993c1254c015c78e8575e09bb9"
server_key_version: "encryptionkey@0"
I20260620 07:12:33.227051 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:33.231837 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:33.232457 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:33.232682 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:33.233233 14105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:33.234138 14105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:33.234184 14105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.234222 14105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:33.234251 14105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:33.243149 14105 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:41793
I20260620 07:12:33.243176 14232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:41793 every 8 connection(s)
I20260620 07:12:33.244059 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:33.250584 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:33.245978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:52647 (local address 127.3.245.62:34691)
0620 07:12:33.246142 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:33.246147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:33.247311 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 07:12:33.248809 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:33.248812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:33.248866 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:33.248946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:33.250090 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:33.250093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:33.250096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:12:33.250108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:33.250129 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:12:33.250137 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:33.250218 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:33.250340 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 07:12:33.250468 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:12:33.251037 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 14105
I20260620 07:12:33.251121 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:12:33.251413 4052 external_mini_cluster.cc:1428] Setting key 1416d4778bbbeb3ee7b8376bc0c8ae7e
I20260620 07:12:33.251587 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:33.246496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34691 (local address 127.3.245.3:52647)
0620 07:12:33.247150 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:33.247191 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:33.248482 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:33.249113 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:33.249120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:33.249214 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:33.249826 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:12:33.249827 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:33.249869 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:12:33.249940 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:33.250215 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:33.250221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:33.250225 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:12:33.250389 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 07:12:33.250660 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260620 07:12:33.253037 14233 heartbeater.cc:347] Connected to a master server at 127.3.245.62:34691
I20260620 07:12:33.253285 14233 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:33.253762 14233 heartbeater.cc:511] Master 127.3.245.62:34691 requested a full tablet report, sending...
I20260620 07:12:33.254994 13782 ts_manager.cc:194] Registered new tserver with Master: f74cc9aa8eaa4407b7d839fd4712555f (127.3.245.3:41793)
I20260620 07:12:33.265661 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 07:12:33.268661 13822 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:12:33.267707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.3.245.62:34691)
0620 07:12:33.267859 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:33.267862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:33.267960 (+ 98us) server_negotiation.cc:407] Connection header received
0620 07:12:33.268067 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:33.268069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:33.268103 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:33.268170 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:33.268452 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48248: BlockingRecv error: recv got EOF from 127.0.0.1:48248 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:12:33.268716 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13843
I20260620 07:12:33.275583 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13974
I20260620 07:12:33.281610 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 14105
I20260620 07:12:33.286943 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 13752
2026-06-20T07:12:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (831 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:33 dist-test-slave-ldfr krb5kdc[14243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:33 dist-test-slave-ldfr krb5kdc[14243](info): set up 2 sockets
Jun 20 07:12:33 dist-test-slave-ldfr krb5kdc[14243](info): commencing operation
krb5kdc: starting...
W20260620 07:12:34.385813 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:34 dist-test-slave-ldfr krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:12:34.516302 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45253
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:34.631873 14256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:34.632172 14256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:34.632220 14256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:34.637250 14256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:12:34.637368 14256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:34.637425 14256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:34.637465 14256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:12:34.637501 14256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:12:34.644140 14256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:34.645403 14256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:34.646502 14256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:34.655962 14262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:34.655965 14264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:34.656136 14256 server_base.cc:1034] running on GCE node
W20260620 07:12:34.656320 14261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:34.656996 14256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:34.658198 14256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:34.659427 14256 hybrid_clock.cc:648] HybridClock initialized: now 1781939554659396 us; error 42 us; skew 500 ppm
Jun 20 07:12:34 dist-test-slave-ldfr krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:34.668715 14256 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:12:34.669823 14256 webserver.cc:460] Webserver started at http://127.3.245.62:44809/ using document root <none> and password file <none>
I20260620 07:12:34.670495 14256 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:34.670785 14256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:34.671070 14256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:34.673291 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "3b308118405140e590b3f308ce6f6c3c"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "dffd13bc77d838aaf0826e9dee4b83ef"
server_key_iv: "ec0ed14b24663db2a20977d32f7af639"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.673875 14256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "3b308118405140e590b3f308ce6f6c3c"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "dffd13bc77d838aaf0826e9dee4b83ef"
server_key_iv: "ec0ed14b24663db2a20977d32f7af639"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.677811 14256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 07:12:34.680395 14271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:34.681833 14256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:12:34.682014 14256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "3b308118405140e590b3f308ce6f6c3c"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "dffd13bc77d838aaf0826e9dee4b83ef"
server_key_iv: "ec0ed14b24663db2a20977d32f7af639"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.682152 14256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:34.697280 14256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:34.700675 14256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:34.700920 14256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:34.709003 14256 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45253
I20260620 07:12:34.709013 14323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45253 every 8 connection(s)
I20260620 07:12:34.710063 14256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:12:34.713161 14324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:34.713629 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 14256
I20260620 07:12:34.713733 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:12:34.714030 4052 external_mini_cluster.cc:1428] Setting key f5d739965df21280daa844b7c461a9c5
I20260620 07:12:34.720638 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: Bootstrap starting.
I20260620 07:12:34.723310 14324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: Neither blocks nor log segments found. Creating new log.
Jun 20 07:12:34 dist-test-slave-ldfr krb5kdc[14243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:34.724299 14324 log.cc:824] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:34.726867 14324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: No bootstrap required, opened a new log
I20260620 07:12:34.729923 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:34.715495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.3.245.62:45253)
0620 07:12:34.716001 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:34.716026 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:34.716072 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:12:34.716977 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:34.716999 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:34.717380 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:34.717781 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:34.718697 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:34.719788 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:34.720841 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:34.721396 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:34.721581 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:34.724665 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:34.724673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:34.724699 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:34.724739 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:34.726770 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:34.727327 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:34.727333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:34.727337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:34.727455 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:34.727840 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:34.727844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:34.727846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:34.728263 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:34.728462 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:34.728818 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 07:12:34.729141 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":153,"threads_started":1}
I20260620 07:12:34.731515 14324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } }
I20260620 07:12:34.731804 14324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:34.731868 14324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b308118405140e590b3f308ce6f6c3c, State: Initialized, Role: FOLLOWER
I20260620 07:12:34.732337 14324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } }
I20260620 07:12:34.732465 14324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:12:34.732515 14324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:12:34.732591 14324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:34.733616 14324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } }
I20260620 07:12:34.733985 14324 leader_election.cc:302] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b308118405140e590b3f308ce6f6c3c; no voters:
I20260620 07:12:34.734337 14324 leader_election.cc:288] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:12:34.734462 14329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:34.734720 14329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [term 1 LEADER]: Becoming Leader. State: Replica: 3b308118405140e590b3f308ce6f6c3c, State: Running, Role: LEADER
I20260620 07:12:34.735013 14329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } }
I20260620 07:12:34.735662 14324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:12:34.736694 14331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b308118405140e590b3f308ce6f6c3c. Latest consensus state: current_term: 1 leader_uuid: "3b308118405140e590b3f308ce6f6c3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } } }
I20260620 07:12:34.736831 14331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:34.737128 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b308118405140e590b3f308ce6f6c3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b308118405140e590b3f308ce6f6c3c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45253 } } }
I20260620 07:12:34.737226 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c [sys.catalog]: This master's current role is: LEADER
I20260620 07:12:34.737450 14338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:12:34.740180 14338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:12:34.745848 14338 catalog_manager.cc:1347] Generated new cluster ID: 7994f36dcfc147ec8939c65d882482fd
I20260620 07:12:34.745931 14338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:12:34.758791 14338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:12:34.760077 14338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:12:34.766417 14338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: Generated new TSK 0
I20260620 07:12:34.767307 14338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:12:34.797895 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:34.914332 14350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:34.914651 14350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:34.914727 14350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:34.918861 14350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:34.918968 14350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:34.919065 14350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:12:34.924715 14350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:34.925863 14350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:34.926945 14350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:34.937563 14358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:34.937557 14355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:34.937557 14356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:34.937736 14350 server_base.cc:1034] running on GCE node
I20260620 07:12:34.938247 14350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:34.939023 14350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:34.940232 14350 hybrid_clock.cc:648] HybridClock initialized: now 1781939554940212 us; error 39 us; skew 500 ppm
Jun 20 07:12:34 dist-test-slave-ldfr krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:34.943388 14350 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:12:34.944275 14350 webserver.cc:460] Webserver started at http://127.3.245.1:34219/ using document root <none> and password file <none>
I20260620 07:12:34.944952 14350 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:34.945031 14350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:34.945295 14350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:34.947503 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "3091585376004aa48b02a9e67da57aa0"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "1e68ebb12c73d8182e50f62ebb70bcb9"
server_key_iv: "23fabaed00cfea05889c71a15a5896d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.948094 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "3091585376004aa48b02a9e67da57aa0"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "1e68ebb12c73d8182e50f62ebb70bcb9"
server_key_iv: "23fabaed00cfea05889c71a15a5896d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.952040 14350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:12:34.954821 14365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:34.956169 14350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:12:34.956369 14350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "3091585376004aa48b02a9e67da57aa0"
format_stamp: "Formatted at 2026-06-20 07:12:34 on dist-test-slave-ldfr"
server_key: "1e68ebb12c73d8182e50f62ebb70bcb9"
server_key_iv: "23fabaed00cfea05889c71a15a5896d6"
server_key_version: "encryptionkey@0"
I20260620 07:12:34.956501 14350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:34.979141 14350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:34.983110 14350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:34.983371 14350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:34.984052 14350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:34.985131 14350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:34.985239 14350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:34.985358 14350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:34.985399 14350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:34.996050 14350 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:35681
I20260620 07:12:34.996153 14478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:35681 every 8 connection(s)
I20260620 07:12:34.997164 14350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:12:35.005649 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 14350
I20260620 07:12:35.005801 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:12:35.006098 4052 external_mini_cluster.cc:1428] Setting key 3442c19b0659f232047adc04915a9693
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:35.012332 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:34.999160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48887 (local address 127.3.245.62:45253)
0620 07:12:34.999314 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:34.999318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.000228 (+ 910us) server_negotiation.cc:407] Connection header received
0620 07:12:35.001619 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.001623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.001682 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.001785 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:35.002961 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.003514 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.004740 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.004978 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.005094 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.008510 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:35.008513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:35.008515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:35.008538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:35.010244 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.010788 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.010791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.010793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.010882 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.011229 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.011232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.011233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.011432 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:35.011539 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.012048 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.012172 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:12:35.013260 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:34.999581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45253 (local address 127.3.245.1:48887)
0620 07:12:35.000071 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.000114 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.001404 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.001973 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.001980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.002333 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:35.002799 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.002812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.003646 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.003649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.004558 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.004569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.005139 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.005142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.005316 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.006293 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:35.006310 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:35.008314 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:35.010385 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.010392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.010408 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.010674 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.010974 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.010978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.010980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.011124 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.011538 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:35.011545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:35.011802 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.012011 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.012349 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260620 07:12:35.014532 14479 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45253
I20260620 07:12:35.014788 14479 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:35.015291 14479 heartbeater.cc:511] Master 127.3.245.62:45253 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:12:35.016925 14288 ts_manager.cc:194] Registered new tserver with Master: 3091585376004aa48b02a9e67da57aa0 (127.3.245.1:35681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:12:35.018230 14288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:48887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:12:35.033818 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:35.153980 14484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:35.154258 14484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:35.154302 14484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:35.158588 14484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:35.158661 14484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:35.158752 14484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:12:35.164984 14484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:35.166150 14484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:35.167193 14484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:35.178773 14489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:35.181505 14490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:35.184736 14492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:35.184900 14484 server_base.cc:1034] running on GCE node
I20260620 07:12:35.185511 14484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:35.186401 14484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:35.187564 14484 hybrid_clock.cc:648] HybridClock initialized: now 1781939555187547 us; error 34 us; skew 500 ppm
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:35.190474 14484 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:12:35.191372 14484 webserver.cc:460] Webserver started at http://127.3.245.2:36753/ using document root <none> and password file <none>
I20260620 07:12:35.191890 14484 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:35.191951 14484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:35.192134 14484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:35.194280 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "882e1f059a744817809476e8198ab0f5"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "7d178831856ef2ffcf9dc72fd52cec2e"
server_key_iv: "4345dcab9dfdbd68a702b37e0f96cbcd"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.194790 14484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "882e1f059a744817809476e8198ab0f5"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "7d178831856ef2ffcf9dc72fd52cec2e"
server_key_iv: "4345dcab9dfdbd68a702b37e0f96cbcd"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.198604 14484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 07:12:35.201115 14499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.202400 14484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:12:35.202497 14484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "882e1f059a744817809476e8198ab0f5"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "7d178831856ef2ffcf9dc72fd52cec2e"
server_key_iv: "4345dcab9dfdbd68a702b37e0f96cbcd"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.202586 14484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:35.215960 14484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:35.219427 14484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:35.219604 14484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:35.220160 14484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:35.221117 14484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:35.221164 14484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.221206 14484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:35.221221 14484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.231182 14484 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:40465
I20260620 07:12:35.231209 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:40465 every 8 connection(s)
I20260620 07:12:35.232121 14484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:12:35.233915 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 14484
I20260620 07:12:35.234074 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:12:35.234473 4052 external_mini_cluster.cc:1428] Setting key 573da21baf44d8d5e5b7ed05ff06c604
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:12:35.249344 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.234105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:60825 (local address 127.3.245.62:45253)
0620 07:12:35.234274 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.234280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.235285 (+ 1005us) server_negotiation.cc:407] Connection header received
0620 07:12:35.237172 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.237179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.237246 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.237349 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:35.239108 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.239671 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.241644 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.241874 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.241985 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.245214 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:35.245218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:35.245220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:35.245249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:35.246763 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.247356 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.247359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.247361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.247451 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.247826 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.247830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.247832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.248079 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:35.248178 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.248928 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.249078 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:12:35.250157 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.234565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45253 (local address 127.3.245.2:60825)
0620 07:12:35.235101 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.235159 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.236826 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.237547 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.237554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.238113 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:35.238893 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.238922 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.239836 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.239843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.241474 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.241490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.242071 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.242079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.242344 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.243083 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:35.243112 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:35.245000 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:35.246904 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.246910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.246935 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.247230 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.247558 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.247561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.247563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.247722 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.248219 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:35.248226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:35.248597 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.248828 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.249123 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
I20260620 07:12:35.251472 14613 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45253
I20260620 07:12:35.251876 14613 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:35.252425 14613 heartbeater.cc:511] Master 127.3.245.62:45253 requested a full tablet report, sending...
I20260620 07:12:35.253679 14288 ts_manager.cc:194] Registered new tserver with Master: 882e1f059a744817809476e8198ab0f5 (127.3.245.2:40465)
I20260620 07:12:35.254292 14288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:60825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:12:35.265951 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:12:35.378623 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:12:35.378890 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:12:35.378934 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:12:35.383009 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:12:35.383077 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:12:35.383170 14618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:12:35.388767 14618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:12:35.389834 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:12:35.390844 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:12:35.400961 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:35.401007 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:12:35.401005 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:12:35.401252 14618 server_base.cc:1034] running on GCE node
I20260620 07:12:35.401881 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:12:35.402624 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:12:35.403863 14618 hybrid_clock.cc:648] HybridClock initialized: now 1781939555403837 us; error 38 us; skew 500 ppm
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:12:35.407040 14618 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:12:35.408010 14618 webserver.cc:460] Webserver started at http://127.3.245.3:34437/ using document root <none> and password file <none>
I20260620 07:12:35.408601 14618 fs_manager.cc:359] Metadata directory not provided
I20260620 07:12:35.408769 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:12:35.409011 14618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:12:35.411134 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "5d6dde758a5f4f0f9db4c600a0393b0c"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "0477e5b244677680dc025e7eca4cbbb4"
server_key_iv: "c2779027f61c64fe4b91d1635a5b394c"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.411711 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "5d6dde758a5f4f0f9db4c600a0393b0c"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "0477e5b244677680dc025e7eca4cbbb4"
server_key_iv: "c2779027f61c64fe4b91d1635a5b394c"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.415678 14618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 07:12:35.418536 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.420009 14618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:12:35.420193 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "5d6dde758a5f4f0f9db4c600a0393b0c"
format_stamp: "Formatted at 2026-06-20 07:12:35 on dist-test-slave-ldfr"
server_key: "0477e5b244677680dc025e7eca4cbbb4"
server_key_iv: "c2779027f61c64fe4b91d1635a5b394c"
server_key_version: "encryptionkey@0"
I20260620 07:12:35.420336 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:12:35.447582 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:12:35.450933 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:12:35.451162 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:12:35.451807 14618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:12:35.452760 14618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:12:35.452839 14618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.452910 14618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:12:35.452952 14618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:12:35.462921 14618 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:33891
I20260620 07:12:35.462965 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:33891 every 8 connection(s)
I20260620 07:12:35.463901 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:12:35.473445 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 14618
I20260620 07:12:35.473608 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939555, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:12:35.473958 4052 external_mini_cluster.cc:1428] Setting key 2e5dcf986e4d5caaf6287454e066919e
I20260620 07:12:35.478492 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.465815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:49639 (local address 127.3.245.62:45253)
0620 07:12:35.465957 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.465961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.466847 (+ 886us) server_negotiation.cc:407] Connection header received
0620 07:12:35.468234 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.468237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.468318 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.468420 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:35.469637 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.470180 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.471358 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.471587 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.471700 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.474417 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:35.474422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:35.474425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:35.474455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:35.476182 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.476817 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.476821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.476822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.476911 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.477265 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.477267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.477269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.477481 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:35.477567 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.478220 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.478355 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:12:35.479362 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.466172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45253 (local address 127.3.245.3:49639)
0620 07:12:35.466637 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.466703 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.468020 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.468599 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.468605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.469016 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:12:35.469487 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.469501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.470314 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.470318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.471224 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.471233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.471738 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.471742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.471929 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.472541 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:12:35.472558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:12:35.474182 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:12:35.476309 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.476315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.476328 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.476675 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.477013 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:12:35.477015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:12:35.477017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:12:35.477160 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:12:35.477639 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:12:35.477645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:12:35.477947 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.478123 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.478415 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
I20260620 07:12:35.480664 14747 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45253
I20260620 07:12:35.480944 14747 heartbeater.cc:464] Registering TS with master...
I20260620 07:12:35.481528 14747 heartbeater.cc:511] Master 127.3.245.62:45253 requested a full tablet report, sending...
I20260620 07:12:35.482715 14288 ts_manager.cc:194] Registered new tserver with Master: 5d6dde758a5f4f0f9db4c600a0393b0c (127.3.245.3:33891)
I20260620 07:12:35.483381 14288 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:49639
I20260620 07:12:35.488188 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:12:35.498744 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.490463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.3.245.62:45253)
0620 07:12:35.490652 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.490659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.490814 (+ 155us) server_negotiation.cc:407] Connection header received
0620 07:12:35.490999 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.491003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.491049 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.491141 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:12:35.491991 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.492558 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.493548 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.493793 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.493922 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.495426 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:12:35.495435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:12:35.495437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:12:35.495463 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:12:35.496986 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.497440 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.497443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.497445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.497531 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:12:35.497809 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:12:35.497821 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:12:35.497822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:12:35.498051 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:12:35.498195 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.498442 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.498561 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 07:12:35.501873 14288 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:12:35.505081 14288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:12:35.520792 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.513964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:33891 (local address 127.0.0.1:36296)
0620 07:12:35.514925 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.514977 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.515169 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.515729 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.515734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.515760 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:35.515893 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.515900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517504 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.517510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.519336 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.519344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520147 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.520265 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520302 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.520439 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.520575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":197,"thread_start_us":71,"threads_started":1}
I20260620 07:12:35.520896 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.513981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:40465 (local address 127.0.0.1:43712)
0620 07:12:35.514947 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.514992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.515123 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.515886 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.515889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.515904 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:35.516009 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.516015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517637 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.517640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.519347 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.519359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520463 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.520565 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520584 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.520676 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.520734 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":403,"thread_start_us":109,"threads_started":1}
I20260620 07:12:35.521003 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.515025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.3.245.1:35681)
0620 07:12:35.515484 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.515493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.515731 (+ 238us) server_negotiation.cc:407] Connection header received
0620 07:12:35.515977 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.515985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.516179 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.516300 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:35.516688 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517472 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.519164 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520034 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520160 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520332 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.520722 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.520835 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":122,"threads_started":1}
I20260620 07:12:35.521261 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.514073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.3.245.3:33891)
0620 07:12:35.514372 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.514383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.515077 (+ 694us) server_negotiation.cc:407] Connection header received
0620 07:12:35.515385 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.515390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.515548 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.516038 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:35.516193 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517348 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.519450 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520017 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520147 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520306 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.521009 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.521120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
I20260620 07:12:35.521327 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.514771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.3.245.2:40465)
0620 07:12:35.515200 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.515212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.515236 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:12:35.515335 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.515340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.515548 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.515708 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:35.516128 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517358 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.519510 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520328 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520457 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520603 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.520719 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.520835 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":127,"threads_started":1}
I20260620 07:12:35.521739 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.514696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:35681 (local address 127.0.0.1:41478)
0620 07:12:35.515641 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.515658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.515816 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.516447 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.516449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.516464 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:35.516535 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.516539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.517594 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.517597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.519037 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.519047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.520410 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.520417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.520515 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.520542 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.521480 (+ 938us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.521589 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":162,"thread_start_us":100,"threads_started":1}
I20260620 07:12:35.524171 14547 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e (DEFAULT_TABLE table=test-table [id=9091fa8fbe7f47d89dcfbbfc023747c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:35.524735 14681 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e (DEFAULT_TABLE table=test-table [id=9091fa8fbe7f47d89dcfbbfc023747c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:35.524736 14413 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e (DEFAULT_TABLE table=test-table [id=9091fa8fbe7f47d89dcfbbfc023747c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:12:35.525372 14547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:35.525785 14681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:35.525785 14413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:12:35.531903 14764 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Bootstrap starting.
I20260620 07:12:35.533079 14765 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: Bootstrap starting.
I20260620 07:12:35.534178 14766 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Bootstrap starting.
I20260620 07:12:35.535060 14765 tablet_bootstrap.cc:654] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:35.535519 14764 tablet_bootstrap.cc:654] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:35.536099 14765 log.cc:824] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:35.536336 14764 log.cc:824] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:35.536521 14766 tablet_bootstrap.cc:654] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Neither blocks nor log segments found. Creating new log.
I20260620 07:12:35.537451 14766 log.cc:824] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:12:35.538327 14764 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: No bootstrap required, opened a new log
I20260620 07:12:35.538518 14765 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: No bootstrap required, opened a new log
I20260620 07:12:35.538666 14764 ts_tablet_manager.cc:1397] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 07:12:35.538690 14765 ts_tablet_manager.cc:1397] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:12:35.539641 14766 tablet_bootstrap.cc:492] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: No bootstrap required, opened a new log
I20260620 07:12:35.539858 14766 ts_tablet_manager.cc:1397] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:12:35.542768 14766 raft_consensus.cc:348] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.542790 14764 raft_consensus.cc:348] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.543038 14766 raft_consensus.cc:374] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:35.543053 14764 raft_consensus.cc:374] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:35.543102 14766 raft_consensus.cc:729] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6dde758a5f4f0f9db4c600a0393b0c, State: Initialized, Role: FOLLOWER
I20260620 07:12:35.543102 14764 raft_consensus.cc:729] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 882e1f059a744817809476e8198ab0f5, State: Initialized, Role: FOLLOWER
I20260620 07:12:35.543123 14765 raft_consensus.cc:348] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.543373 14765 raft_consensus.cc:374] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:12:35.543427 14765 raft_consensus.cc:729] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3091585376004aa48b02a9e67da57aa0, State: Initialized, Role: FOLLOWER
I20260620 07:12:35.543603 14764 consensus_queue.cc:260] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.543661 14766 consensus_queue.cc:260] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.543776 14765 consensus_queue.cc:260] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.544698 14765 ts_tablet_manager.cc:1428] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260620 07:12:35.544698 14764 ts_tablet_manager.cc:1428] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 07:12:35.544719 14766 ts_tablet_manager.cc:1428] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 07:12:35.545075 14747 heartbeater.cc:503] Master 127.3.245.62:45253 was elected leader, sending a full tablet report...
I20260620 07:12:35.545269 14479 heartbeater.cc:503] Master 127.3.245.62:45253 was elected leader, sending a full tablet report...
I20260620 07:12:35.546061 14613 heartbeater.cc:503] Master 127.3.245.62:45253 was elected leader, sending a full tablet report...
I20260620 07:12:35.575934 14771 raft_consensus.cc:482] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:12:35.576193 14771 raft_consensus.cc:504] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.577579 14771 leader_election.cc:288] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 882e1f059a744817809476e8198ab0f5 (127.3.245.2:40465), 3091585376004aa48b02a9e67da57aa0 (127.3.245.1:35681)
I20260620 07:12:35.581136 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.577747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:40465 (local address 127.3.245.3:44907)
0620 07:12:35.577876 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.577894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.577992 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.578277 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.578281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.578298 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:35.578422 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.578428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.579158 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.579162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.580008 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.580017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.580687 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.580690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.580798 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.580826 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.580919 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.581002 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":22}
I20260620 07:12:35.581136 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.577869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:44907 (local address 127.3.245.2:40465)
0620 07:12:35.577997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.578004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.578025 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:12:35.578101 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.578104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.578163 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.578268 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:35.578544 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.579045 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.580135 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.580542 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.580691 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.580757 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.580944 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.581009 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:12:35.581755 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.578081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:56845 (local address 127.3.245.1:35681)
0620 07:12:35.578203 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.578207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.578325 (+ 118us) server_negotiation.cc:407] Connection header received
0620 07:12:35.578477 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.578484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.578527 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.578613 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:12:35.578866 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.579353 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.580518 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.581024 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.581148 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.581214 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.581566 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.581649 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:12:35.581766 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.577840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:35681 (local address 127.3.245.3:56845)
0620 07:12:35.578229 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.578250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:12:35.578361 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:12:35.578623 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:12:35.578626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:12:35.578641 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:12:35.578759 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.578764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.579462 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.579469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.580313 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:12:35.580322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.581169 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.581173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:12:35.581425 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.581447 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:12:35.581549 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:12:35.581621 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":40,"thread_start_us":170,"threads_started":1}
I20260620 07:12:35.581845 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9e2f2ec7d9430e85831aa7eb16cd4e" candidate_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "882e1f059a744817809476e8198ab0f5" is_pre_election: true
I20260620 07:12:35.582201 14567 raft_consensus.cc:2454] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6dde758a5f4f0f9db4c600a0393b0c in term 0.
I20260620 07:12:35.582497 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9e2f2ec7d9430e85831aa7eb16cd4e" candidate_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3091585376004aa48b02a9e67da57aa0" is_pre_election: true
I20260620 07:12:35.582733 14635 leader_election.cc:302] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6dde758a5f4f0f9db4c600a0393b0c, 882e1f059a744817809476e8198ab0f5; no voters:
I20260620 07:12:35.582774 14433 raft_consensus.cc:2454] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6dde758a5f4f0f9db4c600a0393b0c in term 0.
I20260620 07:12:35.583003 14771 raft_consensus.cc:2781] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:12:35.583113 14771 raft_consensus.cc:482] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:12:35.583176 14771 raft_consensus.cc:3037] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:35.584195 14771 raft_consensus.cc:504] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.584631 14771 leader_election.cc:288] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [CANDIDATE]: Term 1 election: Requested vote from peers 882e1f059a744817809476e8198ab0f5 (127.3.245.2:40465), 3091585376004aa48b02a9e67da57aa0 (127.3.245.1:35681)
I20260620 07:12:35.585032 14567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9e2f2ec7d9430e85831aa7eb16cd4e" candidate_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "882e1f059a744817809476e8198ab0f5"
I20260620 07:12:35.585049 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9e2f2ec7d9430e85831aa7eb16cd4e" candidate_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3091585376004aa48b02a9e67da57aa0"
I20260620 07:12:35.585147 14433 raft_consensus.cc:3037] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:35.585177 14567 raft_consensus.cc:3037] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:12:35.586263 14433 raft_consensus.cc:2454] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6dde758a5f4f0f9db4c600a0393b0c in term 1.
I20260620 07:12:35.586504 14567 raft_consensus.cc:2454] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6dde758a5f4f0f9db4c600a0393b0c in term 1.
I20260620 07:12:35.586596 14637 leader_election.cc:302] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3091585376004aa48b02a9e67da57aa0, 5d6dde758a5f4f0f9db4c600a0393b0c; no voters:
I20260620 07:12:35.586768 14771 raft_consensus.cc:2781] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:12:35.587001 14771 raft_consensus.cc:686] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 1 LEADER]: Becoming Leader. State: Replica: 5d6dde758a5f4f0f9db4c600a0393b0c, State: Running, Role: LEADER
I20260620 07:12:35.587339 14771 consensus_queue.cc:237] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } }
I20260620 07:12:35.590536 14288 catalog_manager.cc:5495] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d6dde758a5f4f0f9db4c600a0393b0c (127.3.245.3). New cstate: current_term: 1 leader_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d6dde758a5f4f0f9db4c600a0393b0c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 } health_report { overall_health: UNKNOWN } } }
I20260620 07:12:35.620999 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.617232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.3.245.3:33891)
0620 07:12:35.617433 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:12:35.617437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:12:35.617455 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:12:35.617593 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:12:35.617596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:12:35.617674 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:12:35.617790 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:12:35.618015 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.618519 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.619389 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:12:35.619608 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:12:35.619708 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:12:35.619962 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:12:35.620438 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:12:35.620542 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:12:35.620680 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:12:35.620810 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 07:12:35.627676 14433 raft_consensus.cc:1261] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 1 FOLLOWER]: Refusing update from remote peer 5d6dde758a5f4f0f9db4c600a0393b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:35.627676 14567 raft_consensus.cc:1261] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 1 FOLLOWER]: Refusing update from remote peer 5d6dde758a5f4f0f9db4c600a0393b0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:12:35.628494 14771 consensus_queue.cc:1035] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "882e1f059a744817809476e8198ab0f5" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:35.628767 14774 consensus_queue.cc:1035] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3091585376004aa48b02a9e67da57aa0" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:12:35.636786 14777 mvcc.cc:205] Tried to move back new op lower bound from 7298824419842990080 to 7298824419688402944. Current Snapshot: MvccSnapshot[applied={T|T < 7298824419842990080}]
I20260620 07:12:35.637568 14778 mvcc.cc:205] Tried to move back new op lower bound from 7298824419842990080 to 7298824419688402944. Current Snapshot: MvccSnapshot[applied={T|T < 7298824419842990080}]
I20260620 07:12:35.640172 14779 mvcc.cc:205] Tried to move back new op lower bound from 7298824419842990080 to 7298824419688402944. Current Snapshot: MvccSnapshot[applied={T|T < 7298824419842990080}]
I20260620 07:12:35.646354 14286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:35.646557 14286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:12:35.649178 14286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b308118405140e590b3f308ce6f6c3c: Sending DeleteTablet for 3 replicas of tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e
I20260620 07:12:35.650166 14547 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:43712
I20260620 07:12:35.650224 14413 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:41478
I20260620 07:12:35.650334 14681 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:12:35 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:36296
I20260620 07:12:35.650868 14789 tablet_replica.cc:330] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: stopping tablet replica
I20260620 07:12:35.651180 14788 tablet_replica.cc:330] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0: stopping tablet replica
I20260620 07:12:35.651345 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 14350
I20260620 07:12:35.651675 14788 raft_consensus.cc:2229] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 3091585376004aa48b02a9e67da57aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:35.652197 14789 raft_consensus.cc:2229] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:12:35.652710 14789 raft_consensus.cc:2258] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:12:35.653460 14787 tablet_replica.cc:330] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: stopping tablet replica
I20260620 07:12:35.653791 14787 raft_consensus.cc:2229] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:12:35.654152 14787 raft_consensus.cc:2258] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:12:35.659464 14275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:35681 (error 108)
I20260620 07:12:35.660768 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.660346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:35681 (local address 127.0.0.1:41492)
0620 07:12:35.660563 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.660674 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 07:12:35.660969 14275 catalog_manager.cc:4584] TS 3091585376004aa48b02a9e67da57aa0 (127.3.245.1:35681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e: Network error: Client connection negotiation failed: client connection to 127.3.245.1:35681: connect: Connection refused (error 111)
I20260620 07:12:35.661155 14789 ts_tablet_manager.cc:1905] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:35.661460 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 14484
I20260620 07:12:35.661821 14787 ts_tablet_manager.cc:1905] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 882e1f059a744817809476e8198ab0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:12:35.665288 14789 ts_tablet_manager.cc:1918] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:12:35.665448 14789 log.cc:1196] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/4d9e2f2ec7d9430e85831aa7eb16cd4e
I20260620 07:12:35.665792 14789 ts_tablet_manager.cc:1939] T 4d9e2f2ec7d9430e85831aa7eb16cd4e P 5d6dde758a5f4f0f9db4c600a0393b0c: Deleting consensus metadata
I20260620 07:12:35.667073 14275 catalog_manager.cc:4841] TS 5d6dde758a5f4f0f9db4c600a0393b0c (127.3.245.3:33891): tablet 4d9e2f2ec7d9430e85831aa7eb16cd4e (table test-table [id=9091fa8fbe7f47d89dcfbbfc023747c5]) successfully deleted
I20260620 07:12:35.669404 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 14618
I20260620 07:12:35.671608 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:12:35.669631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:40465 (local address 127.0.0.1:43714)
0620 07:12:35.671285 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0620 07:12:35.671401 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1528}
I20260620 07:12:35.675511 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 14256
2026-06-20T07:12:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2389 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14793](info): set up 2 sockets
Jun 20 07:12:35 dist-test-slave-ldfr krb5kdc[14793](info): commencing operation
krb5kdc: starting...
W20260620 07:12:36.753722 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:12:36 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:12:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangeradmin_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangeradmin_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangerlookup_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangerlookup_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
I20260620 07:12:36.934058 4052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVl0LDo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/postgres -l logfile start
2026-06-20 07:12:39.538 UTC [14823] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:12:39.538 UTC [14823] LOG: listening on IPv4 address "127.3.245.20", port 35079
2026-06-20 07:12:39.542 UTC [14823] LOG: listening on Unix socket "/tmp/.s.PGSQL.35079"
2026-06-20 07:12:39.562 UTC [14826] LOG: database system was shut down at 2026-06-20 07:12:38 UTC
2026-06-20 07:12:39.566 UTC [14823] LOG: database system is ready to accept connections
I20260620 07:12:40.534255 4052 mini_postgres.cc:96] Postgres bound to 35079
2026-06-20 07:12:40.541 UTC [14834] FATAL: database "slave" does not exist
127.3.245.20:35079 - accepting connections
I20260620 07:12:40.541750 4052 mini_ranger.cc:161] Starting Ranger...
I20260620 07:12:40.558002 4052 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:12:41.057399 4052 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:12:41.057502 4052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin
2026-06-20 07:12:41,350 [I] DB FLAVOR :POSTGRES
2026-06-20 07:12:41,351 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:12:41,351 [I] Checking connection..
2026-06-20 07:12:41,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:12:41,584 [I] Checking connection passed.
2026-06-20 07:12:41,584 [I] --------- Verifying version history table ---------
2026-06-20 07:12:41,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:12:41,828 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:12:41,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:12:42,063 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:12:42,063 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:12:42,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:12:42.311 UTC [14898] WARNING: there is no transaction in progress
2026-06-20 07:12:42,317 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:12:42,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:12:42,567 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:12:42,567 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:12:42,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:12:42,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:12:43,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ldfr.c.gcp-upstream.internal','N') ;"
2026-06-20 07:12:43,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:12:43,594 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:12:43,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:12:43,855 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:12:43,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:12:44,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:12:44,343 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:12:44,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:12:45.368 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.386 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.404 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.421 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.439 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.458 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.515 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.522 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.538 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.549 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.561 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.576 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.587 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.594 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.601 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:45.609 UTC [15006] WARNING: there is no transaction in progress
2026-06-20 07:12:46,275 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:12:46,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:12:46,500 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:12:46,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:12:46,750 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:12:46,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:12:46,997 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:12:46,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:12:47,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:12:47,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ldfr.c.gcp-upstream.internal';"
2026-06-20 07:12:47,719 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:12:47,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:12:47,978 [I] DB_PATCHES have already been applied
I20260620 07:12:47.984675 4052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVl0LDo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:12:47.984753 4052 mini_ranger.cc:193] Using host: 127.3.245.20
I20260620 07:12:47.988353 4052 mini_ranger.cc:227] Ranger admin URL: http://127.3.245.20:60513
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:12:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:12:49.130312 4052 mini_ranger.cc:160] Time spent starting Ranger: real 8.589s user 0.003s sys 0.007s
Jun 20, 2026 7:13:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:13:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 7:13:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:13:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:13:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:13:11.903505 4052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangerkms_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/rangerkms_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.245.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.245.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:13:11 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939591, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:13:11.995102 4052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:13:12.005896 4052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:13:13.009011 4052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:13:13.009125 4052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms
I20260620 07:13:13.009232 4052 mini_ranger_kms.cc:227] Using postgres at 127.3.245.20:35079
2026-06-20 07:13:13,200 [I] DB FLAVOR :POSTGRES
2026-06-20 07:13:13,200 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:13:13,200 [I] Checking connection
2026-06-20 07:13:13,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:13:13,467 [I] connection success
2026-06-20 07:13:13,467 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:13:13,467 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:13:13,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:13:13,696 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:13:13,697 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:13:13,697 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:13:13,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:35079/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:13:13,975 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:13:14.476008 4052 mini_ranger_kms.cc:310] Created kms service
I20260620 07:13:14.660046 4052 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:13:14.713886 4052 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:13:14 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939591, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:15.185724 4052 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:13:15.185889 4052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:13:15.185918 4052 mini_ranger_kms.cc:240] Using host: 127.3.245.20
I20260620 07:13:15.190162 4052 mini_ranger_kms.cc:276] Ranger KMS PID: 15187
I20260620 07:13:15.190327 4052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.245.20:49123
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:13:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:13:17.465744 4052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.471s user 0.016s sys 0.154s
I20260620 07:13:17.465895 4052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:13:17.466028 4052 mini_ranger_kms.cc:401] 127.3.245.20:49123/kms/v1/keys
Jun 20 07:13:17 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:13:17 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
Jun 20 07:13:18 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939597, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
Jun 20 07:13:20 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:13:20.637135 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44399
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:44399
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:49123/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:20.768128 15237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:20.768419 15237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:20.768460 15237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:20.773034 15237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:13:20.773109 15237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:20.773129 15237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:20.773146 15237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:13:20.773164 15237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:13:20.778795 15237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.245.20:49123/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:44399
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:13:20.779913 15237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:20.780969 15237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:13:20.789731 15237 server_base.cc:1034] running on GCE node
W20260620 07:13:20.789930 15243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:20.789958 15245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:20.789989 15242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:20.790642 15237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:20.791798 15237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:20.792961 15237 hybrid_clock.cc:648] HybridClock initialized: now 1781939600792946 us; error 27 us; skew 500 ppm
Jun 20 07:13:20 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939600, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:20.795878 15237 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:13:20.796816 15237 webserver.cc:460] Webserver started at http://127.3.245.62:35559/ using document root <none> and password file <none>
I20260620 07:13:20.797338 15237 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:20.797390 15237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:20.797559 15237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:13:20 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939600, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:20.845870 15237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "5dcb615efab44164bf5c9dd013d36986"
format_stamp: "Formatted at 2026-06-20 07:13:20 on dist-test-slave-ldfr"
server_key: "df9ffb8824ebab9cced52a0cd53e4471"
server_key_iv: "69bba0eb6cefa9c18459c15cdaebcd63"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:20.846733 15237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "5dcb615efab44164bf5c9dd013d36986"
format_stamp: "Formatted at 2026-06-20 07:13:20 on dist-test-slave-ldfr"
server_key: "df9ffb8824ebab9cced52a0cd53e4471"
server_key_iv: "69bba0eb6cefa9c18459c15cdaebcd63"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:20.851989 15237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:13:20.870004 15252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:20.871485 15237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:20.871681 15237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "5dcb615efab44164bf5c9dd013d36986"
format_stamp: "Formatted at 2026-06-20 07:13:20 on dist-test-slave-ldfr"
server_key: "df9ffb8824ebab9cced52a0cd53e4471"
server_key_iv: "69bba0eb6cefa9c18459c15cdaebcd63"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:20.871832 15237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:20.897043 15237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:20.900735 15237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:20.900972 15237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:20.908797 15237 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:44399
I20260620 07:13:20.908859 15304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:44399 every 8 connection(s)
I20260620 07:13:20.909842 15237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:13:20.913089 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:20.916247 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 15237
I20260620 07:13:20.916406 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:13:20.916806 4052 external_mini_cluster.cc:1428] Setting key f5b5d1a20ec181b6e4ff0026ff146e5b
I20260620 07:13:20.920251 15305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: Bootstrap starting.
I20260620 07:13:20.923125 15305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:20.923933 15305 log.cc:824] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:20.926468 15305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: No bootstrap required, opened a new log
Jun 20 07:13:20 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:13:20.930320 15305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } }
I20260620 07:13:20.930609 15305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:20.930678 15305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcb615efab44164bf5c9dd013d36986, State: Initialized, Role: FOLLOWER
I20260620 07:13:20.931200 15305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } }
I20260620 07:13:20.931376 15305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:13:20.931470 15305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:13:20.931578 15305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:20.932775 15305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } }
I20260620 07:13:20.933166 15305 leader_election.cc:302] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dcb615efab44164bf5c9dd013d36986; no voters:
I20260620 07:13:20.933279 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:20.918467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.3.245.62:44399)
0620 07:13:20.918965 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:20.918980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:20.919019 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:13:20.919750 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:20.919770 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:20.920128 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:20.920483 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:20.921493 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:20.922551 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:20.923553 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:20.924161 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:20.924338 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:20.928055 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:20.928065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:20.928083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:20.928115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:20.930082 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:20.930609 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:20.930632 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:20.930642 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:20.930785 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:20.931085 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:20.931090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:20.931092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:20.931607 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:20.931801 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:20.932094 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 07:13:20.932388 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20260620 07:13:20.933555 15305 leader_election.cc:288] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:13:20.933943 15311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:13:20.934170 15311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [term 1 LEADER]: Becoming Leader. State: Replica: 5dcb615efab44164bf5c9dd013d36986, State: Running, Role: LEADER
I20260620 07:13:20.934531 15311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } }
I20260620 07:13:20.934938 15305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:13:20.935705 15311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dcb615efab44164bf5c9dd013d36986. Latest consensus state: current_term: 1 leader_uuid: "5dcb615efab44164bf5c9dd013d36986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } } }
I20260620 07:13:20.935930 15311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:20.936102 15312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dcb615efab44164bf5c9dd013d36986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcb615efab44164bf5c9dd013d36986" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44399 } } }
I20260620 07:13:20.936188 15312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986 [sys.catalog]: This master's current role is: LEADER
I20260620 07:13:20.936368 15319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:13:20.939239 15319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:13:20.939715 15305 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:13:20.948768 15319 catalog_manager.cc:1347] Generated new cluster ID: 71f83a46604a450787b4e2a75f581f08
I20260620 07:13:20.948851 15319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:13:20.974328 15319 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:13:20.975385 15319 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:13:20.984660 15319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: Generated new TSK 0
I20260620 07:13:20.985534 15319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:13:21.670986 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:21.793614 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:21.793905 15350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:21.793949 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:21.798134 15350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:21.798197 15350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:21.798285 15350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:13:21.803731 15350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.245.20:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:13:21.804904 15350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:21.805930 15350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:21.815976 15355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:21.816718 15356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:21.817631 15350 server_base.cc:1034] running on GCE node
W20260620 07:13:21.816002 15358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:21.818471 15350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:21.819155 15350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:21.820350 15350 hybrid_clock.cc:648] HybridClock initialized: now 1781939601820312 us; error 52 us; skew 500 ppm
Jun 20 07:13:21 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:21.823173 15350 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:13:21.824224 15350 webserver.cc:460] Webserver started at http://127.3.245.1:34453/ using document root <none> and password file <none>
I20260620 07:13:21.824856 15350 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:21.824928 15350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:21.825136 15350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:13:21 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:21.841843 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "a18d340fe99847639927aa74f71fb34b"
format_stamp: "Formatted at 2026-06-20 07:13:21 on dist-test-slave-ldfr"
server_key: "6cb6a8ec3a9847aa23e4e749ea0e6ba5"
server_key_iv: "b75ee7989f7884884f3c203a5b28004c"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:21.842586 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "a18d340fe99847639927aa74f71fb34b"
format_stamp: "Formatted at 2026-06-20 07:13:21 on dist-test-slave-ldfr"
server_key: "6cb6a8ec3a9847aa23e4e749ea0e6ba5"
server_key_iv: "b75ee7989f7884884f3c203a5b28004c"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:21.847834 15350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:13:21.864415 15365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:21.866137 15350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:21.866283 15350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "a18d340fe99847639927aa74f71fb34b"
format_stamp: "Formatted at 2026-06-20 07:13:21 on dist-test-slave-ldfr"
server_key: "6cb6a8ec3a9847aa23e4e749ea0e6ba5"
server_key_iv: "b75ee7989f7884884f3c203a5b28004c"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:21.866416 15350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:21.896726 15350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:21.900400 15350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:21.900660 15350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:21.901324 15350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:21.902302 15350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:21.902411 15350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:21.902544 15350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:21.902631 15350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:21.919942 15350 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:40929
I20260620 07:13:21.921166 15350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:13:21.924113 15480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:40929 every 8 connection(s)
I20260620 07:13:21.928799 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 15350
I20260620 07:13:21.928941 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:13:21.929277 4052 external_mini_cluster.cc:1428] Setting key 469c82c610b26d8009cecd63c024418f
Jun 20 07:13:21 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939601, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:13:21.941836 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:21.926756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44399 (local address 127.3.245.1:35367)
0620 07:13:21.927199 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:21.927253 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:21.928084 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:21.928678 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:21.928690 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:21.929060 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:21.929724 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:21.929741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:21.930718 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:21.930721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:21.932135 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:21.932149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:21.932835 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:21.932842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:21.933061 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:21.933800 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:21.933827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:21.935591 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:21.938155 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:21.938164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:21.938186 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:21.938555 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:21.938928 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:21.938933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:21.938939 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:21.939140 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:21.939691 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:21.939701 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:21.940093 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 07:13:21.940293 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 07:13:21.940674 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:13:21.943009 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:21.926335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:35367 (local address 127.3.245.62:44399)
0620 07:13:21.926672 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:21.926687 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:21.927370 (+ 683us) server_negotiation.cc:407] Connection header received
0620 07:13:21.928286 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:21.928290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:21.928347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:21.928470 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:21.930015 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:21.930614 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:21.932267 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:21.932649 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:21.932758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:21.935810 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:21.935817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:21.935823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:21.935855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:21.938009 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:21.938693 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:21.938699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:21.938704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:21.938809 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:21.939273 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:21.939279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:21.939282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:21.939552 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:21.942565 (+ 3013us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:21.942650 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 07:13:21.942790 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":125,"threads_started":1}
I20260620 07:13:21.944065 15481 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44399
I20260620 07:13:21.944312 15481 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:21.944943 15481 heartbeater.cc:511] Master 127.3.245.62:44399 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:13:21.946653 15269 ts_manager.cc:194] Registered new tserver with Master: a18d340fe99847639927aa74f71fb34b (127.3.245.1:40929)
I20260620 07:13:21.948202 15269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:35367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:13:21.968101 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:13:22.131367 15487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:22.131803 15487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:22.131878 15487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:22.138862 15487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:22.138964 15487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:22.139056 15487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:13:22.147461 15487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.245.20:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:13:22.149154 15487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:22.150576 15487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:22.163911 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:22.163955 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:22.163947 15498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:22.165236 15487 server_base.cc:1034] running on GCE node
I20260620 07:13:22.165751 15487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:22.166569 15487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:22.167755 15487 hybrid_clock.cc:648] HybridClock initialized: now 1781939602167723 us; error 55 us; skew 500 ppm
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:22.172251 15487 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:13:22.173509 15487 webserver.cc:460] Webserver started at http://127.3.245.2:34655/ using document root <none> and password file <none>
I20260620 07:13:22.174302 15487 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:22.174407 15487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:22.174669 15487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:22.191540 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "6076d4e27b9046a7857e6e0a05b1645b"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "ddf2e7598a20e58d56e34e600106a5d6"
server_key_iv: "9d2601cd03445813502c0c0a7476ed8f"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.192795 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "6076d4e27b9046a7857e6e0a05b1645b"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "ddf2e7598a20e58d56e34e600106a5d6"
server_key_iv: "9d2601cd03445813502c0c0a7476ed8f"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.206511 15487 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260620 07:13:22.229066 15505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.231120 15487 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260620 07:13:22.231271 15487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "6076d4e27b9046a7857e6e0a05b1645b"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "ddf2e7598a20e58d56e34e600106a5d6"
server_key_iv: "9d2601cd03445813502c0c0a7476ed8f"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.231420 15487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:13:22.254828 15487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:22.258868 15487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:22.259079 15487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:22.259820 15487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:22.260936 15487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:22.261003 15487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.261056 15487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:22.261087 15487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.272794 15487 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:37609
I20260620 07:13:22.272838 15618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:37609 every 8 connection(s)
I20260620 07:13:22.274001 15487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:13:22.278674 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 15487
I20260620 07:13:22.278811 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:13:22.279109 4052 external_mini_cluster.cc:1428] Setting key f7d8cd73a00acfa77cc9644a2b2c8ffc
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:13:22.292892 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.279548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:37501 (local address 127.3.245.62:44399)
0620 07:13:22.279678 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.279682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.279697 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:13:22.279742 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.279745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.279793 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.279892 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:22.281605 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.282464 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.284281 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.284667 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.284770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.288060 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:22.288064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:22.288067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:22.288099 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:22.290046 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.290788 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.290793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.290795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.290890 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.291353 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.291357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.291359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.291607 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:22.291685 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.292490 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.292670 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:13:22.294754 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.277308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44399 (local address 127.3.245.2:37501)
0620 07:13:22.277933 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.277997 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.279085 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.280115 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.280126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.280676 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:22.281417 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.281440 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.282668 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.282671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.284120 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.284134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.285020 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.285025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.285234 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.286003 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:22.286029 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:22.287830 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:22.290214 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:22.290221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:22.290242 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:22.290639 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:22.291014 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:22.291017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:22.291020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:22.291227 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:22.291789 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:22.291797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:22.292175 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.293160 (+ 985us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.293561 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":156,"threads_started":1}
I20260620 07:13:22.296907 15619 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44399
I20260620 07:13:22.297183 15619 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:22.297828 15619 heartbeater.cc:511] Master 127.3.245.62:44399 requested a full tablet report, sending...
I20260620 07:13:22.299000 15269 ts_manager.cc:194] Registered new tserver with Master: 6076d4e27b9046a7857e6e0a05b1645b (127.3.245.2:37609)
I20260620 07:13:22.299780 15269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:37501
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:13:22.325695 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:49123/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:13:22.468964 15624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:13:22.469295 15624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:13:22.469373 15624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:13:22.474944 15624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:13:22.475037 15624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:13:22.475138 15624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:13:22.481303 15624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.245.20:49123/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:13:22.482429 15624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:13:22.483541 15624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:13:22.493419 15630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:22.493551 15632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:13:22.493419 15629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:13:22.494940 15624 server_base.cc:1034] running on GCE node
I20260620 07:13:22.495424 15624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:13:22.496248 15624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:13:22.499539 15624 hybrid_clock.cc:648] HybridClock initialized: now 1781939602499462 us; error 93 us; skew 500 ppm
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:13:22.502666 15624 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:13:22.503644 15624 webserver.cc:460] Webserver started at http://127.3.245.3:45003/ using document root <none> and password file <none>
I20260620 07:13:22.504248 15624 fs_manager.cc:359] Metadata directory not provided
I20260620 07:13:22.504335 15624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:13:22.504724 15624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:22.520262 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "9f46b7e7584e4886ba9d662f8956a218"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "2ed764f3a36b485a5c6b0d378fb3014f"
server_key_iv: "7518458aa18d5985d7793c535a61b5d4"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.521203 15624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "9f46b7e7584e4886ba9d662f8956a218"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "2ed764f3a36b485a5c6b0d378fb3014f"
server_key_iv: "7518458aa18d5985d7793c535a61b5d4"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.526085 15624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 07:13:22.542855 15639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.544690 15624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:13:22.544837 15624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "9f46b7e7584e4886ba9d662f8956a218"
format_stamp: "Formatted at 2026-06-20 07:13:22 on dist-test-slave-ldfr"
server_key: "2ed764f3a36b485a5c6b0d378fb3014f"
server_key_iv: "7518458aa18d5985d7793c535a61b5d4"
server_key_version: "kuduclusterkey@0"
I20260620 07:13:22.545002 15624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:13:22.579689 15624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:13:22.583597 15624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:13:22.583817 15624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:13:22.584589 15624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:13:22.585695 15624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:13:22.585762 15624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.585824 15624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:13:22.585856 15624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:13:22.598732 15624 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:37587
I20260620 07:13:22.600165 15752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:37587 every 8 connection(s)
I20260620 07:13:22.600277 15624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:13:22.604596 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 15624
I20260620 07:13:22.604761 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:13:22.605111 4052 external_mini_cluster.cc:1428] Setting key 04fd4ed9894162707641271da5992b65
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:13:22.622807 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.604166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:33581 (local address 127.3.245.62:44399)
0620 07:13:22.604313 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.604318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.606575 (+ 2257us) server_negotiation.cc:407] Connection header received
0620 07:13:22.607819 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.607823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.607876 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.607988 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:22.609570 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.610403 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.612119 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.612478 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.612560 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.616980 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:22.616984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:22.616987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:22.617016 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:22.619490 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.620868 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.620872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.620874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.620964 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.621413 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.621417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.621419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.621658 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:22.621749 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.622509 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.622641 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:13:22.624866 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.604717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44399 (local address 127.3.245.3:33581)
0620 07:13:22.606426 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.606488 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.607559 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.608305 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.608315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.608717 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:13:22.609398 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.609418 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.610565 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.610569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.611975 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.611985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.613935 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.613945 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.614183 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.614926 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:13:22.614949 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:13:22.616759 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:13:22.620304 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:22.620310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:22.620329 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:22.620742 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:22.621075 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:13:22.621078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:13:22.621080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:13:22.621289 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:13:22.621823 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:13:22.621830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:13:22.622205 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.623363 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.623727 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"thread_start_us":167,"threads_started":1}
I20260620 07:13:22.626423 15753 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44399
I20260620 07:13:22.626688 15753 heartbeater.cc:464] Registering TS with master...
I20260620 07:13:22.627190 15753 heartbeater.cc:511] Master 127.3.245.62:44399 requested a full tablet report, sending...
I20260620 07:13:22.628265 15269 ts_manager.cc:194] Registered new tserver with Master: 9f46b7e7584e4886ba9d662f8956a218 (127.3.245.3:37587)
I20260620 07:13:22.629027 15269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:33581
I20260620 07:13:22.632771 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:13:22.649935 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.635114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.3.245.62:44399)
0620 07:13:22.635237 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.635241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.641042 (+ 5801us) server_negotiation.cc:407] Connection header received
0620 07:13:22.641277 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.641281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.641334 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.641388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:13:22.642492 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.643316 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.644505 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.644911 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.645009 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.646873 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:13:22.646877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:13:22.646880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:13:22.646909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:13:22.648297 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.648738 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.648742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.648744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.648821 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:13:22.649085 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:13:22.649090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:13:22.649091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:13:22.649266 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:13:22.649399 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.649631 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.649758 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:13:22.653033 15269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:13:22.656252 15269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:13:22.673297 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.665290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:37587 (local address 127.0.0.1:52414)
0620 07:13:22.665674 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.665719 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.665837 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.667540 (+ 1703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.667550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.667580 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:22.667752 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.667764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.669367 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.669371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.670815 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.670826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.672751 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.672755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.672876 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.672919 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.673005 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.673120 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":109,"threads_started":1}
I20260620 07:13:22.673975 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.665992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.3.245.3:37587)
0620 07:13:22.666281 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.666294 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.666318 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:13:22.666392 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.666397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.666604 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.667924 (+ 1320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:22.668084 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.669244 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.671726 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.672578 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.673507 (+ 929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.673683 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.673736 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.673831 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
I20260620 07:13:22.677160 15686 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bad4f221e483482ce330ec75cd839 (DEFAULT_TABLE table=test-table [id=6e5d57ee15ce43d4bab71136c0c20ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:22.678194 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bad4f221e483482ce330ec75cd839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:22.681371 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.667026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37609 (local address 127.0.0.1:41148)
0620 07:13:22.668237 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.668250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.668330 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.669488 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.669491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.669507 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:22.669590 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.669595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.670956 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.670959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.672182 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.672191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.673833 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.673837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.673923 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.673945 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.674012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.681213 (+ 7201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":90,"threads_started":1}
I20260620 07:13:22.682459 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.667430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.3.245.2:37609)
0620 07:13:22.668559 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.668569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.668931 (+ 362us) server_negotiation.cc:407] Connection header received
0620 07:13:22.669006 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.669011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.669202 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.669356 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:22.669719 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.670835 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.672342 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.673207 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.673306 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.682102 (+ 8796us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.682171 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.682287 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":149,"threads_started":1}
I20260620 07:13:22.685535 15535 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bad4f221e483482ce330ec75cd839 (DEFAULT_TABLE table=test-table [id=6e5d57ee15ce43d4bab71136c0c20ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:22.687219 15535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bad4f221e483482ce330ec75cd839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:22.691250 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.667897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:40929 (local address 127.0.0.1:60206)
0620 07:13:22.668752 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.668764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.668844 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.685959 (+ 17115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.685963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.685982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:22.686156 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.686166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.687612 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.687616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.689714 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.689725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.690834 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.690838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.690938 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.690967 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.691021 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.691097 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":77,"threads_started":1}
I20260620 07:13:22.692185 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.668084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.3.245.1:40929)
0620 07:13:22.685247 (+ 17163us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.685255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.685283 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:13:22.685362 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.685368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.685581 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.685787 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:22.686310 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.687480 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.689879 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.690689 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.691742 (+ 1053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.691894 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.691947 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.692041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17039,"thread_start_us":99,"threads_started":1}
I20260620 07:13:22.695490 15414 tablet_service.cc:1467] Processing CreateTablet for tablet 2c7bad4f221e483482ce330ec75cd839 (DEFAULT_TABLE table=test-table [id=6e5d57ee15ce43d4bab71136c0c20ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:13:22.696666 15414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c7bad4f221e483482ce330ec75cd839. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:13:22.697245 15772 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Bootstrap starting.
I20260620 07:13:22.698064 15771 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Bootstrap starting.
I20260620 07:13:22.701171 15772 tablet_bootstrap.cc:654] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:22.702030 15772 log.cc:824] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:22.704315 15771 tablet_bootstrap.cc:654] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:22.705233 15774 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Bootstrap starting.
I20260620 07:13:22.705649 15771 log.cc:824] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:22.707556 15774 tablet_bootstrap.cc:654] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Neither blocks nor log segments found. Creating new log.
I20260620 07:13:22.708353 15774 log.cc:824] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:13:22.709136 15772 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: No bootstrap required, opened a new log
I20260620 07:13:22.709340 15772 ts_tablet_manager.cc:1397] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 07:13:22.710372 15774 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: No bootstrap required, opened a new log
I20260620 07:13:22.710531 15774 ts_tablet_manager.cc:1397] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:13:22.713240 15772 raft_consensus.cc:348] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.713507 15772 raft_consensus.cc:374] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:22.713550 15772 raft_consensus.cc:729] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6076d4e27b9046a7857e6e0a05b1645b, State: Initialized, Role: FOLLOWER
I20260620 07:13:22.714077 15772 consensus_queue.cc:260] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.715132 15772 ts_tablet_manager.cc:1428] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:13:22.715054 15774 raft_consensus.cc:348] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.715332 15774 raft_consensus.cc:374] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:22.715386 15774 raft_consensus.cc:729] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a18d340fe99847639927aa74f71fb34b, State: Initialized, Role: FOLLOWER
I20260620 07:13:22.715499 15619 heartbeater.cc:503] Master 127.3.245.62:44399 was elected leader, sending a full tablet report...
I20260620 07:13:22.716223 15774 consensus_queue.cc:260] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.717523 15774 ts_tablet_manager.cc:1428] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 07:13:22.717943 15481 heartbeater.cc:503] Master 127.3.245.62:44399 was elected leader, sending a full tablet report...
I20260620 07:13:22.721419 15771 tablet_bootstrap.cc:492] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: No bootstrap required, opened a new log
I20260620 07:13:22.721655 15771 ts_tablet_manager.cc:1397] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20260620 07:13:22.726210 15771 raft_consensus.cc:348] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.726527 15771 raft_consensus.cc:374] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:13:22.726599 15771 raft_consensus.cc:729] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f46b7e7584e4886ba9d662f8956a218, State: Initialized, Role: FOLLOWER
I20260620 07:13:22.727108 15771 consensus_queue.cc:260] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.727859 15753 heartbeater.cc:503] Master 127.3.245.62:44399 was elected leader, sending a full tablet report...
I20260620 07:13:22.729164 15771 ts_tablet_manager.cc:1428] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20260620 07:13:22.776388 15620 tablet.cc:2369] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:13:22 dist-test-slave-ldfr krb5kdc[14793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939602, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
W20260620 07:13:22.854300 15754 tablet.cc:2369] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:13:22.927306 15482 tablet.cc:2369] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:13:22.930501 15778 raft_consensus.cc:482] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:13:22.930687 15778 raft_consensus.cc:504] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.932008 15778 leader_election.cc:288] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f46b7e7584e4886ba9d662f8956a218 (127.3.245.3:37587), 6076d4e27b9046a7857e6e0a05b1645b (127.3.245.2:37609)
I20260620 07:13:22.947649 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.933109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:39685 (local address 127.3.245.2:37609)
0620 07:13:22.933240 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.933245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.940185 (+ 6940us) server_negotiation.cc:407] Connection header received
0620 07:13:22.940367 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.940370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.940418 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.940464 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:22.940840 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.941587 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.946342 (+ 4755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.946918 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.947002 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.947064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.947428 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.947510 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:13:22.948380 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.932980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:37609 (local address 127.3.245.1:39685)
0620 07:13:22.940081 (+ 7101us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.940097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.940253 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.940581 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.940584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.940598 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:22.940715 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.940721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.941738 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.941741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.946202 (+ 4461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.946212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.947183 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.947187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.947280 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.947302 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.948132 (+ 830us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.948214 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7029,"thread_start_us":92,"threads_started":1}
I20260620 07:13:22.949409 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bad4f221e483482ce330ec75cd839" candidate_uuid: "a18d340fe99847639927aa74f71fb34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6076d4e27b9046a7857e6e0a05b1645b" is_pre_election: true
I20260620 07:13:22.949747 15572 raft_consensus.cc:2454] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a18d340fe99847639927aa74f71fb34b in term 0.
I20260620 07:13:22.950297 15368 leader_election.cc:302] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6076d4e27b9046a7857e6e0a05b1645b, a18d340fe99847639927aa74f71fb34b; no voters:
I20260620 07:13:22.950685 15778 raft_consensus.cc:2781] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:13:22.950768 15778 raft_consensus.cc:482] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:13:22.950809 15778 raft_consensus.cc:3037] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:22.952153 15778 raft_consensus.cc:504] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.952531 15778 leader_election.cc:288] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [CANDIDATE]: Term 1 election: Requested vote from peers 9f46b7e7584e4886ba9d662f8956a218 (127.3.245.3:37587), 6076d4e27b9046a7857e6e0a05b1645b (127.3.245.2:37609)
I20260620 07:13:22.953261 15572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bad4f221e483482ce330ec75cd839" candidate_uuid: "a18d340fe99847639927aa74f71fb34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6076d4e27b9046a7857e6e0a05b1645b"
I20260620 07:13:22.953394 15572 raft_consensus.cc:3037] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:22.954710 15572 raft_consensus.cc:2454] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a18d340fe99847639927aa74f71fb34b in term 1.
I20260620 07:13:22.955191 15368 leader_election.cc:302] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6076d4e27b9046a7857e6e0a05b1645b, a18d340fe99847639927aa74f71fb34b; no voters:
I20260620 07:13:22.955417 15778 raft_consensus.cc:2781] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:13:22.955637 15778 raft_consensus.cc:686] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [term 1 LEADER]: Becoming Leader. State: Replica: a18d340fe99847639927aa74f71fb34b, State: Running, Role: LEADER
I20260620 07:13:22.956023 15778 consensus_queue.cc:237] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } }
I20260620 07:13:22.956843 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.933511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:54197 (local address 127.3.245.3:37587)
0620 07:13:22.933643 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:22.933648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:22.936179 (+ 2531us) server_negotiation.cc:407] Connection header received
0620 07:13:22.938128 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:22.938133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:22.938219 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:22.938271 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:13:22.938710 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.939467 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.945861 (+ 6394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.946466 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.946542 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.946626 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:22.956559 (+ 9933us) server_negotiation.cc:299] Negotiation successful
0620 07:13:22.956710 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:13:22.957875 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:22.932515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:37587 (local address 127.3.245.1:54197)
0620 07:13:22.936030 (+ 3515us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:22.936046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:13:22.936498 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:13:22.938430 (+ 1932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:13:22.938433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:13:22.938453 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:13:22.938562 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.938569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.939644 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.939646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.945485 (+ 5839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:13:22.945498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:13:22.947410 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:13:22.947413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:13:22.947488 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:22.956401 (+ 8913us) client_negotiation.cc:769] Sending connection context
0620 07:13:22.957518 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0620 07:13:22.957725 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3442,"mutex_wait_us":7,"thread_start_us":147,"threads_started":1}
I20260620 07:13:22.959141 15688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bad4f221e483482ce330ec75cd839" candidate_uuid: "a18d340fe99847639927aa74f71fb34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f46b7e7584e4886ba9d662f8956a218" is_pre_election: true
I20260620 07:13:22.959455 15688 raft_consensus.cc:2454] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a18d340fe99847639927aa74f71fb34b in term 0.
I20260620 07:13:22.959797 15689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c7bad4f221e483482ce330ec75cd839" candidate_uuid: "a18d340fe99847639927aa74f71fb34b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f46b7e7584e4886ba9d662f8956a218"
I20260620 07:13:22.959916 15689 raft_consensus.cc:3037] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:13:22.959908 15269 catalog_manager.cc:5495] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b reported cstate change: term changed from 0 to 1, leader changed from <none> to a18d340fe99847639927aa74f71fb34b (127.3.245.1). New cstate: current_term: 1 leader_uuid: "a18d340fe99847639927aa74f71fb34b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a18d340fe99847639927aa74f71fb34b" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 40929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 } health_report { overall_health: UNKNOWN } } }
I20260620 07:13:22.961229 15689 raft_consensus.cc:2454] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a18d340fe99847639927aa74f71fb34b in term 1.
I20260620 07:13:23.039919 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:23.034543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.3.245.1:40929)
0620 07:13:23.034915 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 07:13:23.034920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:13:23.034935 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:13:23.035045 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:13:23.035049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:13:23.035115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:13:23.035216 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:13:23.035608 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.036432 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.037706 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:13:23.038062 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:13:23.038157 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:13:23.038577 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:13:23.039143 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:13:23.039255 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:13:23.039678 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 07:13:23.039740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260620 07:13:23.048038 15572 raft_consensus.cc:1261] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 1 FOLLOWER]: Refusing update from remote peer a18d340fe99847639927aa74f71fb34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:23.048509 15689 raft_consensus.cc:1261] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 1 FOLLOWER]: Refusing update from remote peer a18d340fe99847639927aa74f71fb34b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:13:23.049074 15778 consensus_queue.cc:1035] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6076d4e27b9046a7857e6e0a05b1645b" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:23.049592 15784 consensus_queue.cc:1035] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f46b7e7584e4886ba9d662f8956a218" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:13:23.060704 15787 mvcc.cc:205] Tried to move back new op lower bound from 7298824614074322944 to 7298824613711523840. Current Snapshot: MvccSnapshot[applied={T|T < 7298824614074322944}]
I20260620 07:13:23.067977 15788 mvcc.cc:205] Tried to move back new op lower bound from 7298824614074322944 to 7298824613711523840. Current Snapshot: MvccSnapshot[applied={T|T < 7298824614074322944}]
I20260620 07:13:23.070688 15268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:23.070874 15268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:13:23.072906 15789 mvcc.cc:205] Tried to move back new op lower bound from 7298824614074322944 to 7298824613711523840. Current Snapshot: MvccSnapshot[applied={T|T < 7298824614074322944}]
I20260620 07:13:23.074700 15268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dcb615efab44164bf5c9dd013d36986: Sending DeleteTablet for 3 replicas of tablet 2c7bad4f221e483482ce330ec75cd839
I20260620 07:13:23.076047 15686 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c7bad4f221e483482ce330ec75cd839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:23 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:52414
I20260620 07:13:23.076596 15414 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c7bad4f221e483482ce330ec75cd839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:23 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60206
I20260620 07:13:23.077037 15535 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c7bad4f221e483482ce330ec75cd839 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:13:23 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:41148
I20260620 07:13:23.077469 15800 tablet_replica.cc:330] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: stopping tablet replica
I20260620 07:13:23.077796 15800 raft_consensus.cc:2229] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:23.078125 15800 raft_consensus.cc:2258] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:23.079224 15798 tablet_replica.cc:330] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: stopping tablet replica
I20260620 07:13:23.079514 15799 tablet_replica.cc:330] T 2c7bad4f221e483482ce330ec75cd839 P a18d340fe99847639927aa74f71fb34b: stopping tablet replica
I20260620 07:13:23.079592 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 15350
I20260620 07:13:23.080982 15798 raft_consensus.cc:2229] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:13:23.081318 15798 raft_consensus.cc:2258] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:13:23.084457 15800 ts_tablet_manager.cc:1905] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:23.085824 15798 ts_tablet_manager.cc:1905] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:13:23.087452 15800 ts_tablet_manager.cc:1918] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:23.087566 15800 log.cc:1196] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/2c7bad4f221e483482ce330ec75cd839
I20260620 07:13:23.087911 15800 ts_tablet_manager.cc:1939] T 2c7bad4f221e483482ce330ec75cd839 P 6076d4e27b9046a7857e6e0a05b1645b: Deleting consensus metadata
W20260620 07:13:23.089006 15254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:40929 (error 108)
I20260620 07:13:23.089252 15798 ts_tablet_manager.cc:1918] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:13:23.089284 15253 catalog_manager.cc:4841] TS 6076d4e27b9046a7857e6e0a05b1645b (127.3.245.2:37609): tablet 2c7bad4f221e483482ce330ec75cd839 (table test-table [id=6e5d57ee15ce43d4bab71136c0c20ec3]) successfully deleted
I20260620 07:13:23.089345 15798 log.cc:1196] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/2c7bad4f221e483482ce330ec75cd839
I20260620 07:13:23.089654 15798 ts_tablet_manager.cc:1939] T 2c7bad4f221e483482ce330ec75cd839 P 9f46b7e7584e4886ba9d662f8956a218: Deleting consensus metadata
I20260620 07:13:23.089962 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 15487
I20260620 07:13:23.092365 15254 catalog_manager.cc:4841] TS 9f46b7e7584e4886ba9d662f8956a218 (127.3.245.3:37587): tablet 2c7bad4f221e483482ce330ec75cd839 (table test-table [id=6e5d57ee15ce43d4bab71136c0c20ec3]) successfully deleted
I20260620 07:13:23.092489 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:13:23.092194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:40929 (local address 127.0.0.1:60230)
0620 07:13:23.092332 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:13:23.092404 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:40929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:13:23.094273 15254 catalog_manager.cc:4584] TS a18d340fe99847639927aa74f71fb34b (127.3.245.1:40929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c7bad4f221e483482ce330ec75cd839: Network error: Client connection negotiation failed: client connection to 127.3.245.1:40929: connect: Connection refused (error 111)
I20260620 07:13:23.097646 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 15624
I20260620 07:13:23.103296 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 15237
I20260620 07:13:23.110463 4052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:13:23.445684 4052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:13:23.445840 4052 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:13:23.782018 4052 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:13:23.782 UTC [14823] LOG: received smart shutdown request
2026-06-20 07:13:23.785 UTC [14823] LOG: background worker "logical replication launcher" (PID 14832) exited with exit code 1
2026-06-20 07:13:23.785 UTC [14827] LOG: shutting down
2026-06-20 07:13:23.950 UTC [14823] LOG: database system is shut down
2026-06-20T07:13:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48391 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:13:24 dist-test-slave-ldfr krb5kdc[15815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:13:24 dist-test-slave-ldfr krb5kdc[15815](info): set up 2 sockets
Jun 20 07:13:24 dist-test-slave-ldfr krb5kdc[15815](info): commencing operation
krb5kdc: starting...
W20260620 07:13:25.112871 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:13:25 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:13:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangeradmin_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangeradmin_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangerlookup_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangerlookup_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
I20260620 07:13:25.287545 4052 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVl0LDo/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/postgres -l logfile start
2026-06-20 07:13:27.717 UTC [15845] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 07:13:27.717 UTC [15845] LOG: listening on IPv4 address "127.3.245.20", port 34041
2026-06-20 07:13:27.720 UTC [15845] LOG: listening on Unix socket "/tmp/.s.PGSQL.34041"
2026-06-20 07:13:27.741 UTC [15848] LOG: database system was shut down at 2026-06-20 07:13:26 UTC
2026-06-20 07:13:27.744 UTC [15845] LOG: database system is ready to accept connections
I20260620 07:13:28.736680 4052 mini_postgres.cc:96] Postgres bound to 34041
2026-06-20 07:13:28.743 UTC [15856] FATAL: database "slave" does not exist
127.3.245.20:34041 - accepting connections
I20260620 07:13:28.744168 4052 mini_ranger.cc:161] Starting Ranger...
I20260620 07:13:28.761912 4052 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 07:13:29.265501 4052 mini_ranger.cc:87] Created ranger Postgres database
I20260620 07:13:29.265631 4052 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin
2026-06-20 07:13:29,561 [I] DB FLAVOR :POSTGRES
2026-06-20 07:13:29,562 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:13:29,562 [I] Checking connection..
2026-06-20 07:13:29,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 07:13:29,801 [I] Checking connection passed.
2026-06-20 07:13:29,801 [I] --------- Verifying version history table ---------
2026-06-20 07:13:29,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:13:30,041 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:13:30,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:13:30,289 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 07:13:30,289 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 07:13:30,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 07:13:30.542 UTC [15920] WARNING: there is no transaction in progress
2026-06-20 07:13:30,549 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 07:13:30,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 07:13:30,807 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 07:13:30,808 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:13:30,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 07:13:31,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 07:13:31,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ldfr.c.gcp-upstream.internal','N') ;"
2026-06-20 07:13:31,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:13:31,828 [I] Table x_portal_user does not exist in database ranger
2026-06-20 07:13:31,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:13:32,107 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 07:13:32,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:13:32,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:13:32,591 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 07:13:32,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 07:13:33.589 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.606 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.625 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.642 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.661 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.682 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.742 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.749 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.763 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.774 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.784 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.799 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.809 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.817 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.825 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:33.832 UTC [16028] WARNING: there is no transaction in progress
2026-06-20 07:13:34,477 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 07:13:34,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 07:13:34,703 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 07:13:34,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 07:13:34,940 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 07:13:34,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 07:13:35,195 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 07:13:35,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 07:13:35,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 07:13:35,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ldfr.c.gcp-upstream.internal';"
2026-06-20 07:13:35,938 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 07:13:35,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 07:13:36,167 [I] DB_PATCHES have already been applied
I20260620 07:13:36.172940 4052 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVl0LDo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 07:13:36.173012 4052 mini_ranger.cc:193] Using host: 127.3.245.20
I20260620 07:13:36.176527 4052 mini_ranger.cc:227] Ranger admin URL: http://127.3.245.20:47601
Jun 20, 2026 7:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:13:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 07:13:37.289506 4052 mini_ranger.cc:160] Time spent starting Ranger: real 8.545s user 0.003s sys 0.007s
Jun 20, 2026 7:13:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:13:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:13:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:13:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:13:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:13:59.130260 4052 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.245.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.245.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangerkms_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.245.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/rangerkms_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.245.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.245.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.245.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/HTTP_127.3.245.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:13:59 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939639, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:13:59.214439 4052 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:13:59.224315 4052 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:14:00.196559 4052 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:14:00.196785 4052 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms
I20260620 07:14:00.196810 4052 mini_ranger_kms.cc:227] Using postgres at 127.3.245.20:34041
2026-06-20 07:14:00,368 [I] DB FLAVOR :POSTGRES
2026-06-20 07:14:00,368 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:14:00,368 [I] Checking connection
2026-06-20 07:14:00,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:14:00,599 [I] connection success
2026-06-20 07:14:00,599 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:14:00,599 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:14:00,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:14:00,837 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:14:00,837 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:14:00,838 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:14:00,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.245.20:34041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:14:01,118 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:14:01.616060 4052 mini_ranger_kms.cc:310] Created kms service
I20260620 07:14:01.830391 4052 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:14:01.881036 4052 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:14:01 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939639, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:02.306128 4052 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:14:02.306281 4052 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVl0LDo/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVl0LDo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:14:02.306303 4052 mini_ranger_kms.cc:240] Using host: 127.3.245.20
I20260620 07:14:02.309919 4052 mini_ranger_kms.cc:276] Ranger KMS PID: 16208
I20260620 07:14:02.310012 4052 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.245.20:35379
Jun 20, 2026 7:14:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:14:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:14:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:14:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:14:04.692593 4052 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.478s user 0.004s sys 0.148s
I20260620 07:14:04.692729 4052 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:14:04.692791 4052 mini_ranger_kms.cc:401] 127.3.245.20:35379/kms/v1/keys
Jun 20 07:14:04 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:14:05 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
Jun 20 07:14:05 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939644, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.245.20@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
Jun 20 07:14:07 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:14:07.784235 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:40035
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:40035
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:07.896371 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:07.896773 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:07.896889 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:07.901279 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:07.901386 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:07.901424 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:07.901465 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:07.901506 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:07.907481 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.245.20:35379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:40035
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:07.908754 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:07.909847 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:07.919389 16264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:07.919601 16259 server_base.cc:1034] running on GCE node
W20260620 07:14:07.919368 16265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:07.919389 16267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:07.920456 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:07.921732 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:07.922931 16259 hybrid_clock.cc:648] HybridClock initialized: now 1781939647922907 us; error 47 us; skew 500 ppm
Jun 20 07:14:07 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:07.931098 16259 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:07.932288 16259 webserver.cc:460] Webserver started at http://127.3.245.62:33705/ using document root <none> and password file <none>
I20260620 07:14:07.932962 16259 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:07.933025 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:07.933265 16259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:14:07 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939647, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:07.990175 16259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "bf0499e0bdb44f2fb77c0d7fc9799514"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-ldfr"
server_key: "73079c3a3a30fc7da987bbe649917ee9"
server_key_iv: "237b5c5fa44225c1f504b51077c704e9"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:07.990969 16259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "bf0499e0bdb44f2fb77c0d7fc9799514"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-ldfr"
server_key: "73079c3a3a30fc7da987bbe649917ee9"
server_key_iv: "237b5c5fa44225c1f504b51077c704e9"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:07.996366 16259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 07:14:08.014920 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:08.016685 16259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:08.016848 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "bf0499e0bdb44f2fb77c0d7fc9799514"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-ldfr"
server_key: "73079c3a3a30fc7da987bbe649917ee9"
server_key_iv: "237b5c5fa44225c1f504b51077c704e9"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:08.017012 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:08.033581 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:08.037243 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:08.037463 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:08.046665 16326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:40035 every 8 connection(s)
I20260620 07:14:08.046679 16259 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:40035
I20260620 07:14:08.048008 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:08.050974 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:08.052440 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16259
I20260620 07:14:08.052582 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:14:08.052935 4052 external_mini_cluster.cc:1428] Setting key 592db610101ad65783ad91cc63bb54c3
I20260620 07:14:08.058468 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Bootstrap starting.
I20260620 07:14:08.061060 16327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:08.062064 16327 log.cc:824] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:08 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:08.064457 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: No bootstrap required, opened a new log
I20260620 07:14:08.068576 16327 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:08.068900 16327 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:08.068961 16327 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0499e0bdb44f2fb77c0d7fc9799514, State: Initialized, Role: FOLLOWER
I20260620 07:14:08.069599 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:08.054449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.3.245.62:40035)
0620 07:14:08.054942 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:08.054963 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:08.055006 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:14:08.055850 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:08.055874 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:08.056302 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:08.056709 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:08.057538 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.058943 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.060188 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:08.060891 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:08.061072 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:08.063930 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:08.063936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:08.063950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:08.063979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:08.066275 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:08.066769 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:08.066774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:08.066779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:08.066884 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:08.067173 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:08.067177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:08.067179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:08.067582 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:08.067808 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:08.068042 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 07:14:08.068351 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":143,"threads_started":1}
I20260620 07:14:08.069545 16327 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:08.069737 16327 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:08.070770 16327 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:08.070892 16327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:08.072082 16327 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:08.072459 16327 leader_election.cc:302] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0499e0bdb44f2fb77c0d7fc9799514; no voters:
I20260620 07:14:08.072858 16327 leader_election.cc:288] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:08.072973 16332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:08.073244 16332 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 LEADER]: Becoming Leader. State: Replica: bf0499e0bdb44f2fb77c0d7fc9799514, State: Running, Role: LEADER
I20260620 07:14:08.073602 16332 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:08.074203 16327 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:08.075332 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } } }
I20260620 07:14:08.075452 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:08.075685 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0499e0bdb44f2fb77c0d7fc9799514. Latest consensus state: current_term: 1 leader_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } } }
I20260620 07:14:08.075778 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:08.076400 16327 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:14:08.078060 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:08.084597 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:08.093122 16341 catalog_manager.cc:1347] Generated new cluster ID: 17bfabb3059b4aa99190f6ebc5a05688
I20260620 07:14:08.093289 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:08.117725 16341 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:08.119232 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:08.132886 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Generated new TSK 0
I20260620 07:14:08.134168 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:14:08.936887 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:09.108600 16372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:09.109196 16372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:09.109344 16372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:09.116963 16372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:09.117197 16372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:09.117359 16372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:09.126526 16372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:09.128134 16372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:09.129731 16372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:09.142195 16382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.145617 16379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.146034 16380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:09.146239 16372 server_base.cc:1034] running on GCE node
I20260620 07:14:09.146770 16372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:09.147629 16372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:09.150924 16372 hybrid_clock.cc:648] HybridClock initialized: now 1781939649150702 us; error 227 us; skew 500 ppm
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:09.154551 16372 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:09.155635 16372 webserver.cc:460] Webserver started at http://127.3.245.1:40555/ using document root <none> and password file <none>
I20260620 07:14:09.156430 16372 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:09.156634 16372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:09.156960 16372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:09.187520 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "fbd77b4497d84cc08b0ba11e02095c8d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "0a514283c4776785c299e1587fbc1fd0"
server_key_iv: "6c3a724f1dda4c7aca434db7d46f56c8"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.188247 16372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "fbd77b4497d84cc08b0ba11e02095c8d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "0a514283c4776785c299e1587fbc1fd0"
server_key_iv: "6c3a724f1dda4c7aca434db7d46f56c8"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.194016 16372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 07:14:09.233151 16389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.235160 16372 fs_manager.cc:725] Time spent opening block manager: real 0.024s user 0.003s sys 0.000s
I20260620 07:14:09.235323 16372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "fbd77b4497d84cc08b0ba11e02095c8d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "0a514283c4776785c299e1587fbc1fd0"
server_key_iv: "6c3a724f1dda4c7aca434db7d46f56c8"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.235479 16372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:09.252753 16372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:09.256256 16372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:09.256523 16372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:09.257198 16372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:09.258309 16372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:09.258415 16372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.258546 16372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:09.258636 16372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.269214 16372 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:34449
I20260620 07:14:09.270437 16372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:09.271000 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16372
I20260620 07:14:09.271126 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:09.271440 4052 external_mini_cluster.cc:1428] Setting key 207b68a9ee5d4dafe8b3cb72559635fa
I20260620 07:14:09.281462 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:34449 every 8 connection(s)
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:09.319991 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.299480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:45195 (local address 127.3.245.62:40035)
0620 07:14:09.302409 (+ 2929us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.302415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.302433 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:14:09.302495 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.302499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.302555 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.302657 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.306811 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.307711 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.311210 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.311601 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.311690 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.315097 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.315102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.315105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:09.315138 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.317091 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:09.317934 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:09.317938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:09.317940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:09.318048 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:09.318508 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:09.318511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:09.318513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:09.318762 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.318850 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.319650 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.319800 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2841,"thread_start_us":78,"threads_started":1}
I20260620 07:14:09.322129 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.298224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.1:45195)
0620 07:14:09.300757 (+ 2533us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.300822 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.301913 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.303156 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.303167 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.303538 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:09.304331 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.304354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.307883 (+ 3529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.307886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.311034 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.311047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.311927 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.311931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.312158 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.312888 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:09.312911 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:09.314837 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:09.317414 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:09.317421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:09.317442 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:09.317811 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:09.318170 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:09.318179 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:09.318181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:09.318388 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:09.318920 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:09.318927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:09.319318 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.320529 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.320952 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"thread_start_us":158,"threads_started":1}
I20260620 07:14:09.324098 16506 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:09.324369 16506 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:09.325019 16506 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:09.326969 16291 ts_manager.cc:194] Registered new tserver with Master: fbd77b4497d84cc08b0ba11e02095c8d (127.3.245.1:34449)
I20260620 07:14:09.328575 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:45195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:14:09.348104 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:14:09.504889 16511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:09.505389 16511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:09.505504 16511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:09.511794 16511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:09.511965 16511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:09.512120 16511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:09.520673 16511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:09.522356 16511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:09.523798 16511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:09.537623 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:09.537984 16511 server_base.cc:1034] running on GCE node
W20260620 07:14:09.538554 16517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.538887 16520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:09.539466 16511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:09.540294 16511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:09.541664 16511 hybrid_clock.cc:648] HybridClock initialized: now 1781939649541591 us; error 88 us; skew 500 ppm
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:09.545539 16511 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:09.546792 16511 webserver.cc:460] Webserver started at http://127.3.245.2:36961/ using document root <none> and password file <none>
I20260620 07:14:09.547608 16511 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:09.547761 16511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:09.548055 16511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:09.570745 16511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "42440624f787475b8f6662c693f47a31"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "72b74b4c2451bf2190baa31b27393908"
server_key_iv: "6a4520858c049e6c053e91eecf13b2d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.571625 16511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "42440624f787475b8f6662c693f47a31"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "72b74b4c2451bf2190baa31b27393908"
server_key_iv: "6a4520858c049e6c053e91eecf13b2d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.579214 16511 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260620 07:14:09.604895 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.606818 16511 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 07:14:09.606976 16511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "42440624f787475b8f6662c693f47a31"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "72b74b4c2451bf2190baa31b27393908"
server_key_iv: "6a4520858c049e6c053e91eecf13b2d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.607122 16511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:09.638833 16511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:09.642856 16511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:09.643141 16511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:09.644073 16511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:09.648298 16511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:09.648389 16511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.648461 16511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:09.648501 16511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.659530 16511 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:43373
I20260620 07:14:09.659555 16640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:43373 every 8 connection(s)
I20260620 07:14:09.660466 16511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:09.661206 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16511
I20260620 07:14:09.661363 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:09.661696 4052 external_mini_cluster.cc:1428] Setting key 589d61660e7b950bba9089310d131322
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:09.683681 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.662556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:60581 (local address 127.3.245.62:40035)
0620 07:14:09.662750 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:09.662754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:09.665152 (+ 2398us) server_negotiation.cc:407] Connection header received
0620 07:14:09.666166 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:09.666170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:09.666225 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:09.666310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:09.667856 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.668684 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.670787 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.671180 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.671314 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.676920 (+ 5606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:09.676927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:09.676934 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:09.676973 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:09.678752 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:09.681815 (+ 3063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:09.681819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:09.681821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:09.681908 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:09.682335 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:09.682338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:09.682340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:09.682570 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:09.682658 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:09.683357 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 07:14:09.683494 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 07:14:09.685563 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:09.663038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.2:60581)
0620 07:14:09.664968 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:09.665031 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:09.665951 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:09.666490 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:09.666498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:09.666897 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:09.667660 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.667684 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.668838 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.668845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.670632 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:09.670647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:09.671383 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:09.671387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:09.671643 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:09.672420 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:09.672448 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:09.676705 (+ 4257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:09.681265 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:09.681272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:09.681293 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:09.681682 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:09.682032 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:09.682035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:09.682037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:09.682233 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:09.682725 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:09.682733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:09.683085 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 07:14:09.684123 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0620 07:14:09.684490 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1678,"thread_start_us":190,"threads_started":1}
I20260620 07:14:09.690240 16641 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:09.690546 16641 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:09.691208 16641 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:09.692474 16291 ts_manager.cc:194] Registered new tserver with Master: 42440624f787475b8f6662c693f47a31 (127.3.245.2:43373)
I20260620 07:14:09.693334 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:60581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:14:09.702008 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
W20260620 07:14:09.863732 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:09.864257 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:09.864368 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:09.870850 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:09.871003 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:09.871136 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:09.879081 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:09.880510 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:09.881968 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:09.894487 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.894627 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:09.894500 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:09.895509 16646 server_base.cc:1034] running on GCE node
I20260620 07:14:09.896063 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:09.896960 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:09.898173 16646 hybrid_clock.cc:648] HybridClock initialized: now 1781939649898142 us; error 44 us; skew 500 ppm
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:09.902112 16646 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:09.903294 16646 webserver.cc:460] Webserver started at http://127.3.245.3:34331/ using document root <none> and password file <none>
I20260620 07:14:09.904045 16646 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:09.904135 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:09.904368 16646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:14:09 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:09.935350 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "771874bec91e45289ea187e0ffefbb9d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "2e5edb1d85e1fc7e2229bd53a3fefa73"
server_key_iv: "c5d4e79b5f5cf753f65a1329311ab1d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.936183 16646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "771874bec91e45289ea187e0ffefbb9d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "2e5edb1d85e1fc7e2229bd53a3fefa73"
server_key_iv: "c5d4e79b5f5cf753f65a1329311ab1d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.943316 16646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 07:14:09.973081 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.974854 16646 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260620 07:14:09.975024 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "771874bec91e45289ea187e0ffefbb9d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "2e5edb1d85e1fc7e2229bd53a3fefa73"
server_key_iv: "c5d4e79b5f5cf753f65a1329311ab1d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:09.975172 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:09.987085 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:09.990959 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:09.991218 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:09.992023 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:09.993798 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:09.993901 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:09.994150 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:09.994231 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.006600 16646 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:41007
I20260620 07:14:10.007580 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:10.008992 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16646
I20260620 07:14:10.009135 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:10.009426 4052 external_mini_cluster.cc:1428] Setting key 0474f137afcbd6540803977989d4d059
I20260620 07:14:10.013661 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:41007 every 8 connection(s)
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939649, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:10.024955 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.010628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:58525 (local address 127.3.245.62:40035)
0620 07:14:10.010780 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:10.010785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:10.011818 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 07:14:10.013208 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:10.013211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:10.013270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:10.013323 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:10.014716 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.015250 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.016908 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.017269 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.017415 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.019972 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:10.019981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:10.019983 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:10.020008 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:10.022102 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.022873 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.022877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.022880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.022964 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.023368 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.023371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.023373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.023693 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:10.023788 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:10.024648 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 07:14:10.024800 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:14:10.026679 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.011192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.3:58525)
0620 07:14:10.011668 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:10.011725 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:10.012954 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:10.013747 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:10.013757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:10.014062 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:10.014565 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.014579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.015377 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.015381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.016781 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.016791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.017471 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.017475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.017668 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.018357 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:10.018374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:10.019791 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:10.022254 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.022261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.022281 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.022699 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.023077 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.023080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.023082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.023258 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.023897 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:10.023904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:10.024307 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 07:14:10.025508 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0620 07:14:10.025829 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260620 07:14:10.028175 16775 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:10.028434 16775 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.029034 16775 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:10.030098 16291 ts_manager.cc:194] Registered new tserver with Master: 771874bec91e45289ea187e0ffefbb9d (127.3.245.3:41007)
I20260620 07:14:10.030685 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:58525
I20260620 07:14:10.037133 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:10.037752 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16372
I20260620 07:14:10.044301 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16511
I20260620 07:14:10.051407 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16646
I20260620 07:14:10.062927 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16259
I20260620 07:14:10.071096 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:40035
--webserver_interface=127.3.245.62
--webserver_port=33705
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:40035
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:10.184969 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:10.185269 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:10.185310 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:10.189368 16778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:10.189435 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:10.189453 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:10.189471 16778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:10.189487 16778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:10.195163 16778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:40035
--ranger_config_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=33705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:10.196260 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:10.197341 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:10.206405 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.206493 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.206519 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:10.206724 16778 server_base.cc:1034] running on GCE node
I20260620 07:14:10.207430 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:10.208664 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:10.209851 16778 hybrid_clock.cc:648] HybridClock initialized: now 1781939650209812 us; error 47 us; skew 500 ppm
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:10.212723 16778 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:10.213634 16778 webserver.cc:460] Webserver started at http://127.3.245.62:33705/ using document root <none> and password file <none>
I20260620 07:14:10.214215 16778 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:10.214309 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:10.268452 16778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:14:10.270865 16794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.272253 16778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:10.272409 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "bf0499e0bdb44f2fb77c0d7fc9799514"
format_stamp: "Formatted at 2026-06-20 07:14:07 on dist-test-slave-ldfr"
server_key: "73079c3a3a30fc7da987bbe649917ee9"
server_key_iv: "237b5c5fa44225c1f504b51077c704e9"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:10.273044 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:10.296036 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:10.299302 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:10.299749 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:10.307021 16778 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:40035
I20260620 07:14:10.307035 16846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:40035 every 8 connection(s)
I20260620 07:14:10.308100 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:10.308844 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16778
I20260620 07:14:10.309265 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:34449
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:40035
--webserver_port=40555
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:10.312552 16847 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:14:10.314244 16847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Bootstrap starting.
I20260620 07:14:10.330627 16847 log.cc:824] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:10.337235 16847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:14:10.337647 16847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Bootstrap complete.
I20260620 07:14:10.341921 16847 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:10.342350 16847 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0499e0bdb44f2fb77c0d7fc9799514, State: Initialized, Role: FOLLOWER
I20260620 07:14:10.342936 16847 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:10.343204 16847 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:10.343307 16847 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:10.343441 16847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:14:10.345464 16847 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:10.345827 16847 leader_election.cc:302] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0499e0bdb44f2fb77c0d7fc9799514; no voters:
I20260620 07:14:10.346248 16847 leader_election.cc:288] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:14:10.346482 16851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:14:10.346772 16851 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [term 2 LEADER]: Becoming Leader. State: Replica: bf0499e0bdb44f2fb77c0d7fc9799514, State: Running, Role: LEADER
I20260620 07:14:10.347149 16851 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } }
I20260620 07:14:10.347654 16847 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:10.348186 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0499e0bdb44f2fb77c0d7fc9799514. Latest consensus state: current_term: 2 leader_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } } }
I20260620 07:14:10.348330 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:10.348817 16855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:10.351142 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0499e0bdb44f2fb77c0d7fc9799514" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 40035 } } }
I20260620 07:14:10.351321 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:10.352650 16855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:10.353295 16855 catalog_manager.cc:1259] Loaded cluster ID: 17bfabb3059b4aa99190f6ebc5a05688
I20260620 07:14:10.353390 16855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:10.353849 16847 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:14:10.358623 16855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:10.359786 16855 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Loaded TSK: 0
I20260620 07:14:10.361299 16855 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:14:10.453336 16849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:10.453716 16849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:10.453781 16849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:10.459837 16849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:10.459997 16849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:10.460148 16849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:10.468058 16849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:34449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=40555
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:10.469444 16849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:10.470711 16849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:10.482957 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.483104 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.482975 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:10.484405 16849 server_base.cc:1034] running on GCE node
I20260620 07:14:10.485047 16849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:10.485865 16849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:10.487066 16849 hybrid_clock.cc:648] HybridClock initialized: now 1781939650487041 us; error 41 us; skew 500 ppm
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:10.490743 16849 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:10.491887 16849 webserver.cc:460] Webserver started at http://127.3.245.1:40555/ using document root <none> and password file <none>
I20260620 07:14:10.492595 16849 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:10.492765 16849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:10.538300 16849 fs_manager.cc:709] Time spent opening directory manager: real 0.024s user 0.003s sys 0.000s
I20260620 07:14:10.549544 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.551469 16849 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260620 07:14:10.551632 16849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "fbd77b4497d84cc08b0ba11e02095c8d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "0a514283c4776785c299e1587fbc1fd0"
server_key_iv: "6c3a724f1dda4c7aca434db7d46f56c8"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:10.552399 16849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:10.583782 16849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:10.593076 16849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:10.593367 16849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:10.594146 16849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:10.595325 16849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:10.595431 16849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.595561 16849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:10.595659 16849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.606319 16849 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:34449
I20260620 07:14:10.607784 16849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:10.611726 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:34449 every 8 connection(s)
I20260620 07:14:10.617322 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 16849
I20260620 07:14:10.617849 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:43373
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:40035
--webserver_port=36961
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:10.630404 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.610651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.1:36637)
0620 07:14:10.612171 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:10.612228 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:10.613384 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:10.615307 (+ 1923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:10.615318 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:10.615670 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:10.616326 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.616348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.618312 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.618315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.619740 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.619751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.620997 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.621001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.621188 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.621821 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:10.621842 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:10.623521 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:10.626170 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.626177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.626192 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.626573 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.627043 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.627049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.627051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.627233 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.628114 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:10.628120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:10.628470 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 07:14:10.628879 (+ 409us) client_negotiation.cc:240] Negotiation successful
0620 07:14:10.629309 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":158,"threads_started":1}
I20260620 07:14:10.632413 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.611897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:36637 (local address 127.3.245.62:40035)
0620 07:14:10.614044 (+ 2147us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:10.614066 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:10.614112 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:14:10.614401 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:10.614416 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:10.614839 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:10.615196 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:10.616537 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.618153 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.619884 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.620602 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.620785 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.623773 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:10.623781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:10.623799 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:10.623833 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:10.626017 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.626747 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.626754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.626759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.626907 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.627353 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.627357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.627359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.627876 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:10.628024 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:10.631019 (+ 2995us) server_negotiation.cc:299] Negotiation successful
0620 07:14:10.631375 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1983,"thread_start_us":110,"threads_started":1}
I20260620 07:14:10.636099 17009 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:10.636437 17009 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.637146 17009 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
W20260620 07:14:10.639513 17009 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:40035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:14:10.639730 17009 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.640002 17009 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:10.640998 17009 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.641239 17009 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
W20260620 07:14:10.642087 17009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:14:10.787897 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:10.788264 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:10.788326 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:10.794490 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:10.794631 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:10.794734 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:10.802484 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:43373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=36961
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:10.804080 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:10.805459 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:10.821163 17021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.822165 17018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:10.822465 17019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:10.823020 17013 server_base.cc:1034] running on GCE node
I20260620 07:14:10.823634 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:10.824481 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:10.825647 17013 hybrid_clock.cc:648] HybridClock initialized: now 1781939650825605 us; error 55 us; skew 500 ppm
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:10.829169 17013 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:10.830258 17013 webserver.cc:460] Webserver started at http://127.3.245.2:36961/ using document root <none> and password file <none>
I20260620 07:14:10.830919 17013 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:10.831012 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:10.854117 17013 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260620 07:14:10.857506 17029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.859133 17013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:10.859272 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "42440624f787475b8f6662c693f47a31"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "72b74b4c2451bf2190baa31b27393908"
server_key_iv: "6a4520858c049e6c053e91eecf13b2d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:10.859982 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:10.885973 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:10.889961 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:10.890239 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:10.890996 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:10.892120 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:10.892184 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:10.892234 17013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:10.892302 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 07:14:10.905910 17013 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:43373
I20260620 07:14:10.907320 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:10.908869 17143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:43373 every 8 connection(s)
I20260620 07:14:10.916204 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17013
I20260620 07:14:10.916798 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.245.20:35379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:41007
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:40035
--webserver_port=34331
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:10 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:10.930298 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.910797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:53031 (local address 127.3.245.62:40035)
0620 07:14:10.911024 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:10.911029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:10.912510 (+ 1481us) server_negotiation.cc:407] Connection header received
0620 07:14:10.913802 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:10.913807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:10.913872 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:10.913924 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:10.915655 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.916498 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.918351 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.918730 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.918823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.922123 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:10.922128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:10.922133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:10.922173 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:10.924106 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.928129 (+ 4023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.928133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.928135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.928248 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:10.928786 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:10.928792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:10.928796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:10.929039 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:10.929141 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:10.929948 (+ 807us) server_negotiation.cc:299] Negotiation successful
0620 07:14:10.930101 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:14:10.931986 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:10.910295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.2:53031)
0620 07:14:10.912322 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:10.912380 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:10.913527 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:10.914379 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:10.914389 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:10.914749 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:10.915443 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.915462 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.916730 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.916734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.918194 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:10.918205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:10.919037 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:10.919040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:10.919243 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:10.919998 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:10.920021 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:10.921888 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:10.926656 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.926664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.926685 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.927962 (+ 1277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.928383 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:10.928389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:10.928392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:10.928653 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:10.929217 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:10.929225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:10.929640 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 07:14:10.930493 (+ 853us) client_negotiation.cc:240] Negotiation successful
0620 07:14:10.930862 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1788,"thread_start_us":742,"threads_started":1}
I20260620 07:14:10.933697 17144 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:10.933959 17144 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.934567 17144 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
W20260620 07:14:10.936417 17144 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:40035 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:14:10.936553 17144 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.936762 17144 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:10.937692 17144 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:10.937865 17144 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
W20260620 07:14:10.938670 17144 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:14:11.089675 17147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:11.090127 17147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:11.090238 17147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:11.096360 17147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:11.096603 17147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:11.096781 17147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:11.104803 17147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.245.20:35379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:41007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=34331
--tserver_master_addrs=127.3.245.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:11.106509 17147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:11.108012 17147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:11.120285 17155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:11.120339 17153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:11.120530 17152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:11.120757 17147 server_base.cc:1034] running on GCE node
I20260620 07:14:11.121413 17147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:11.122273 17147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:11.123471 17147 hybrid_clock.cc:648] HybridClock initialized: now 1781939651123443 us; error 41 us; skew 500 ppm
Jun 20 07:14:11 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:11.127455 17147 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:11.128711 17147 webserver.cc:460] Webserver started at http://127.3.245.3:34331/ using document root <none> and password file <none>
I20260620 07:14:11.129505 17147 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:11.129621 17147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:14:11 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:11.150326 17147 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260620 07:14:11.153455 17163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:11.155225 17147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:14:11.155378 17147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "771874bec91e45289ea187e0ffefbb9d"
format_stamp: "Formatted at 2026-06-20 07:14:09 on dist-test-slave-ldfr"
server_key: "2e5edb1d85e1fc7e2229bd53a3fefa73"
server_key_iv: "c5d4e79b5f5cf753f65a1329311ab1d2"
server_key_version: "kuduclusterkey@0"
I20260620 07:14:11.156215 17147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:11.196411 17147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:11.200836 17147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:11.201081 17147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:11.201949 17147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:11.203104 17147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:11.203176 17147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:11.203255 17147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:11.203341 17147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 07:14:11.216274 17147 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:41007
I20260620 07:14:11.217860 17147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:11.219313 17276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:41007 every 8 connection(s)
I20260620 07:14:11.220850 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17147
Jun 20 07:14:11 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:11.235561 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.223178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.3.245.62:40035)
0620 07:14:11.224079 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.224092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.224508 (+ 416us) server_negotiation.cc:407] Connection header received
0620 07:14:11.224562 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.224565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.224637 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.224725 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:11.225833 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.226659 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.227973 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.228313 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.228409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.230773 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:11.230778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:11.230780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:11.230807 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:11.232404 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.233234 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.233239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.233242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.233345 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.234466 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.234470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.234472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.234769 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:11.234998 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:11.235199 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 07:14:11.235341 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"spinlock_wait_cycles":31488,"thread_start_us":68,"threads_started":1}
I20260620 07:14:11.236881 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.221508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:44931 (local address 127.3.245.62:40035)
0620 07:14:11.221705 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.221710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.222624 (+ 914us) server_negotiation.cc:407] Connection header received
0620 07:14:11.224499 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.224503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.224567 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.224707 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:11.226313 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.227170 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.228993 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.229340 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.229411 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.232393 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:11.232396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:11.232399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:11.232421 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:11.234094 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.234757 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.234761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.234763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.234889 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.235386 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.235392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.235394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.235664 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:11.235754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:11.236543 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 07:14:11.236724 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"spinlock_wait_cycles":1792}
I20260620 07:14:11.238574 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.221965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:40035 (local address 127.3.245.3:44931)
0620 07:14:11.222462 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:11.222516 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:11.224240 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:11.225071 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:11.225081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:11.225437 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:11.226134 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.226156 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.227339 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.227342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:11.228849 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.228860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.229623 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.229627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:11.229871 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.230568 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:11.230598 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:11.232179 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:11.234214 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:11.234219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:11.234235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:11.234613 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:11.235021 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:11.235025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:11.235027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:11.235268 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:11.235844 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:11.235853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:11.236214 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 07:14:11.237040 (+ 826us) client_negotiation.cc:240] Negotiation successful
0620 07:14:11.237421 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":155,"threads_started":1}
I20260620 07:14:11.239986 17277 heartbeater.cc:347] Connected to a master server at 127.3.245.62:40035
I20260620 07:14:11.240278 17277 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:11.240931 17277 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:11.242774 16811 ts_manager.cc:194] Registered new tserver with Master: 771874bec91e45289ea187e0ffefbb9d (127.3.245.3:41007)
I20260620 07:14:11.244040 16811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:44931
Jun 20 07:14:11 dist-test-slave-ldfr krb5kdc[15815](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:11.642379 17009 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:11.642694 17009 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:11.643561 16811 ts_manager.cc:194] Registered new tserver with Master: fbd77b4497d84cc08b0ba11e02095c8d (127.3.245.1:34449)
I20260620 07:14:11.644373 16811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:36637
I20260620 07:14:11.938903 17144 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:11.939191 17144 heartbeater.cc:511] Master 127.3.245.62:40035 requested a full tablet report, sending...
I20260620 07:14:11.940042 16811 ts_manager.cc:194] Registered new tserver with Master: 42440624f787475b8f6662c693f47a31 (127.3.245.2:43373)
I20260620 07:14:11.940933 16811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:53031
I20260620 07:14:11.949882 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:11.961824 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.951528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.3.245.62:40035)
0620 07:14:11.951809 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.951814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.952095 (+ 281us) server_negotiation.cc:407] Connection header received
0620 07:14:11.952187 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.952190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.952242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.952340 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:11.953200 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.954042 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.955226 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.955596 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.955709 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.957389 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:11.957393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:11.957396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:11.957425 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:11.959374 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.959871 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.959875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.959877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.959976 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:11.960301 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:11.960305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:11.960306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:11.960545 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:11.961352 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:11.961509 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 07:14:11.961642 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260620 07:14:11.966632 16811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:11.972455 16811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:11.992666 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.984109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:41007 (local address 127.0.0.1:38628)
0620 07:14:11.986687 (+ 2578us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:11.986742 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:11.986966 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:11.987575 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:11.987581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:11.987608 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:11.987757 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.987765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.989291 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.989294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:11.990711 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.990722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.992084 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.992087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:11.992199 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.992237 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:14:11.992340 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 07:14:11.992470 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":94,"threads_started":1}
I20260620 07:14:11.993273 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.984977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.3.245.3:41007)
0620 07:14:11.985316 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.985323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.986850 (+ 1527us) server_negotiation.cc:407] Connection header received
0620 07:14:11.987159 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.987166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.987389 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.987548 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:11.987906 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.989133 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.990860 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.991715 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.991803 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:11.991985 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:11.993023 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0620 07:14:11.993131 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260620 07:14:11.997092 17211 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d37c3cda741c89031f584b463556c (DEFAULT_TABLE table=test-table [id=ec8c5d25ec8d459abecf58c10d2719c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:11.998479 17211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d37c3cda741c89031f584b463556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:12.006074 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.986290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:43373 (local address 127.0.0.1:54300)
0620 07:14:11.993687 (+ 7397us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:11.993702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:11.993769 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:11.997538 (+ 3769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:11.997541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:11.997559 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:11.997637 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.997643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.002084 (+ 4441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.002087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.003287 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.003299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.005643 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.005647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.005745 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.005769 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:14:12.005839 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:14:12.005920 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7342,"thread_start_us":98,"threads_started":1}
I20260620 07:14:12.006857 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.986688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.3.245.2:43373)
0620 07:14:11.996845 (+ 10157us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.996853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.996880 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:14:11.996979 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.996985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.997207 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.997361 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:12.000745 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.001932 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.003444 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.005312 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.005418 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.005582 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.006608 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.006716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10016,"thread_start_us":105,"threads_started":1}
Jun 20 07:14:12 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:12.010195 17078 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d37c3cda741c89031f584b463556c (DEFAULT_TABLE table=test-table [id=ec8c5d25ec8d459abecf58c10d2719c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:12.011317 17078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d37c3cda741c89031f584b463556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:12.011576 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.985355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.3.245.1:34449)
0620 07:14:11.985626 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:11.985632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:11.991961 (+ 6329us) server_negotiation.cc:407] Connection header received
0620 07:14:11.992077 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:11.992083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:11.992334 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:11.992483 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:11.992914 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.994001 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.009783 (+ 15782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.010673 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.010765 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.010918 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.011303 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.011410 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260620 07:14:12.012176 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:11.984243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34449 (local address 127.0.0.1:47926)
0620 07:14:11.989968 (+ 5725us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:11.989983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:11.990090 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:11.992662 (+ 2572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:11.992665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:11.992685 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:11.992787 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:11.992795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:11.994120 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:11.994126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.009594 (+ 15468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.009607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.011028 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.011032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.011132 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.011156 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:14:12.011932 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 07:14:12.012026 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5654,"thread_start_us":98,"threads_started":1}
I20260620 07:14:12.015349 16934 tablet_service.cc:1467] Processing CreateTablet for tablet 9a0d37c3cda741c89031f584b463556c (DEFAULT_TABLE table=test-table [id=ec8c5d25ec8d459abecf58c10d2719c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:12.016515 16934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a0d37c3cda741c89031f584b463556c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:12.020228 17310 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Bootstrap starting.
I20260620 07:14:12.022730 17310 tablet_bootstrap.cc:654] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:12.023617 17310 log.cc:824] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:12.024945 17313 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Bootstrap starting.
I20260620 07:14:12.026046 17311 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Bootstrap starting.
I20260620 07:14:12.027233 17313 tablet_bootstrap.cc:654] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:12.028086 17313 log.cc:824] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:12.028306 17311 tablet_bootstrap.cc:654] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:12.029176 17311 log.cc:824] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:12.031390 17311 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: No bootstrap required, opened a new log
I20260620 07:14:12.031601 17311 ts_tablet_manager.cc:1397] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:14:12.033682 17310 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: No bootstrap required, opened a new log
I20260620 07:14:12.033905 17310 ts_tablet_manager.cc:1397] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260620 07:14:12.036159 17311 raft_consensus.cc:348] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.036561 17311 raft_consensus.cc:374] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:12.036685 17311 raft_consensus.cc:729] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42440624f787475b8f6662c693f47a31, State: Initialized, Role: FOLLOWER
I20260620 07:14:12.037427 17311 consensus_queue.cc:260] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.038389 17310 raft_consensus.cc:348] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.038683 17310 raft_consensus.cc:374] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:12.038717 17311 ts_tablet_manager.cc:1428] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260620 07:14:12.038779 17310 raft_consensus.cc:729] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771874bec91e45289ea187e0ffefbb9d, State: Initialized, Role: FOLLOWER
I20260620 07:14:12.039148 17144 heartbeater.cc:503] Master 127.3.245.62:40035 was elected leader, sending a full tablet report...
I20260620 07:14:12.039361 17310 consensus_queue.cc:260] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.040376 17310 ts_tablet_manager.cc:1428] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:14:12.040758 17277 heartbeater.cc:503] Master 127.3.245.62:40035 was elected leader, sending a full tablet report...
I20260620 07:14:12.050436 17313 tablet_bootstrap.cc:492] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: No bootstrap required, opened a new log
I20260620 07:14:12.050707 17313 ts_tablet_manager.cc:1397] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20260620 07:14:12.056342 17313 raft_consensus.cc:348] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.056782 17313 raft_consensus.cc:374] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:12.056885 17313 raft_consensus.cc:729] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd77b4497d84cc08b0ba11e02095c8d, State: Initialized, Role: FOLLOWER
I20260620 07:14:12.057576 17313 consensus_queue.cc:260] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.058856 17313 ts_tablet_manager.cc:1428] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:14:12.059315 17009 heartbeater.cc:503] Master 127.3.245.62:40035 was elected leader, sending a full tablet report...
W20260620 07:14:12.113392 17010 tablet.cc:2369] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:12.163008 17145 tablet.cc:2369] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:12.220715 17278 tablet.cc:2369] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 07:14:12 dist-test-slave-ldfr krb5kdc[15815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for HTTP/127.3.245.20@KRBTEST.COM
I20260620 07:14:12.260404 17316 raft_consensus.cc:482] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:12.260658 17316 raft_consensus.cc:504] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.262174 17316 leader_election.cc:288] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 771874bec91e45289ea187e0ffefbb9d (127.3.245.3:41007), fbd77b4497d84cc08b0ba11e02095c8d (127.3.245.1:34449)
I20260620 07:14:12.269642 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.263135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:57955 (local address 127.3.245.1:34449)
0620 07:14:12.263267 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:12.263272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:12.264661 (+ 1389us) server_negotiation.cc:407] Connection header received
0620 07:14:12.264909 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:12.264914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:12.264975 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:12.265035 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:12.265626 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.266428 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.268135 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.268803 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.268897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.268975 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.269393 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.269484 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:14:12.270231 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.262633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34449 (local address 127.3.245.2:57955)
0620 07:14:12.264538 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:12.264554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:12.264758 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:12.265236 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:12.265239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:12.265259 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:12.265433 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.265442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.266673 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.266677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.267965 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.267978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.269089 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.269093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.269215 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.269243 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 07:14:12.269974 (+ 731us) client_negotiation.cc:240] Negotiation successful
0620 07:14:12.270067 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":129,"threads_started":1}
I20260620 07:14:12.271052 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d37c3cda741c89031f584b463556c" candidate_uuid: "42440624f787475b8f6662c693f47a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" is_pre_election: true
I20260620 07:14:12.271394 16959 raft_consensus.cc:2454] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42440624f787475b8f6662c693f47a31 in term 0.
I20260620 07:14:12.272779 17032 leader_election.cc:302] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42440624f787475b8f6662c693f47a31, fbd77b4497d84cc08b0ba11e02095c8d; no voters:
I20260620 07:14:12.273193 17316 raft_consensus.cc:2781] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:12.273288 17316 raft_consensus.cc:482] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:12.273337 17316 raft_consensus.cc:3037] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:12.274857 17316 raft_consensus.cc:504] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.275348 17316 leader_election.cc:288] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [CANDIDATE]: Term 1 election: Requested vote from peers 771874bec91e45289ea187e0ffefbb9d (127.3.245.3:41007), fbd77b4497d84cc08b0ba11e02095c8d (127.3.245.1:34449)
I20260620 07:14:12.276027 16959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d37c3cda741c89031f584b463556c" candidate_uuid: "42440624f787475b8f6662c693f47a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd77b4497d84cc08b0ba11e02095c8d"
I20260620 07:14:12.276180 16959 raft_consensus.cc:3037] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:12.277693 16959 raft_consensus.cc:2454] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42440624f787475b8f6662c693f47a31 in term 1.
I20260620 07:14:12.278154 17032 leader_election.cc:302] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42440624f787475b8f6662c693f47a31, fbd77b4497d84cc08b0ba11e02095c8d; no voters:
I20260620 07:14:12.278822 17316 raft_consensus.cc:2781] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:12.279102 17316 raft_consensus.cc:686] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 1 LEADER]: Becoming Leader. State: Replica: 42440624f787475b8f6662c693f47a31, State: Running, Role: LEADER
I20260620 07:14:12.279531 17316 consensus_queue.cc:237] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } }
I20260620 07:14:12.284076 16810 catalog_manager.cc:5495] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42440624f787475b8f6662c693f47a31 (127.3.245.2). New cstate: current_term: 1 leader_uuid: "42440624f787475b8f6662c693f47a31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42440624f787475b8f6662c693f47a31" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 43373 } health_report { overall_health: HEALTHY } } }
I20260620 07:14:12.287915 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.263988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:56833 (local address 127.3.245.3:41007)
0620 07:14:12.264115 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:12.264121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:12.270916 (+ 6795us) server_negotiation.cc:407] Connection header received
0620 07:14:12.271138 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:12.271142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:12.271198 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:12.271251 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:12.271655 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.272450 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.277890 (+ 5440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.278529 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.278630 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.278745 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.287647 (+ 8902us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.287767 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:14:12.288420 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.263574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:41007 (local address 127.3.245.2:56833)
0620 07:14:12.270829 (+ 7255us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:12.270846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:12.270999 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:12.271406 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:12.271409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:12.271427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:12.271516 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.271522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.275821 (+ 4299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.275824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.277647 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:12.277661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.285721 (+ 8060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.285725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:12.287442 (+ 1717us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.287486 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:14:12.288146 (+ 660us) client_negotiation.cc:240] Negotiation successful
0620 07:14:12.288244 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7190,"thread_start_us":127,"threads_started":1}
I20260620 07:14:12.289443 17231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d37c3cda741c89031f584b463556c" candidate_uuid: "42440624f787475b8f6662c693f47a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771874bec91e45289ea187e0ffefbb9d" is_pre_election: true
I20260620 07:14:12.289779 17231 raft_consensus.cc:2454] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42440624f787475b8f6662c693f47a31 in term 0.
I20260620 07:14:12.290136 17230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a0d37c3cda741c89031f584b463556c" candidate_uuid: "42440624f787475b8f6662c693f47a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "771874bec91e45289ea187e0ffefbb9d"
I20260620 07:14:12.290251 17230 raft_consensus.cc:3037] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:12.291613 17230 raft_consensus.cc:2454] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42440624f787475b8f6662c693f47a31 in term 1.
I20260620 07:14:12.365989 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.360483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.3.245.2:43373)
0620 07:14:12.360655 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:12.360659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:12.360674 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:14:12.360759 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:12.360762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:12.360827 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:12.360947 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:12.361293 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.362062 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.363359 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:12.363739 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:12.363845 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:12.364525 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:12.365159 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:12.365292 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:12.365774 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 07:14:12.365842 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:12.373371 16959 raft_consensus.cc:1261] T 9a0d37c3cda741c89031f584b463556c P fbd77b4497d84cc08b0ba11e02095c8d [term 1 FOLLOWER]: Refusing update from remote peer 42440624f787475b8f6662c693f47a31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:12.374219 17316 consensus_queue.cc:1035] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd77b4497d84cc08b0ba11e02095c8d" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:12.381803 17230 raft_consensus.cc:1261] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 1 FOLLOWER]: Refusing update from remote peer 42440624f787475b8f6662c693f47a31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:12.382611 17316 consensus_queue.cc:1035] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "771874bec91e45289ea187e0ffefbb9d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:12.396432 17326 mvcc.cc:205] Tried to move back new op lower bound from 7298824816112926720 to 7298824815740563456. Current Snapshot: MvccSnapshot[applied={T|T < 7298824816112926720}]
I20260620 07:14:12.413410 16810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:12.413615 16810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:12.416455 16810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf0499e0bdb44f2fb77c0d7fc9799514: Sending DeleteTablet for 3 replicas of tablet 9a0d37c3cda741c89031f584b463556c
I20260620 07:14:12.417443 16934 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d37c3cda741c89031f584b463556c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:12 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:47926
I20260620 07:14:12.418032 17078 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d37c3cda741c89031f584b463556c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:12 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:54300
I20260620 07:14:12.418167 17211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a0d37c3cda741c89031f584b463556c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:12 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:38628
I20260620 07:14:12.419013 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16849
I20260620 07:14:12.423614 17329 mvcc.cc:205] Tried to move back new op lower bound from 7298824816112926720 to 7298824815740563456. Current Snapshot: MvccSnapshot[applied={T|T < 7298824816112926720}]
I20260620 07:14:12.423955 17335 tablet_replica.cc:330] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: stopping tablet replica
I20260620 07:14:12.423954 17336 tablet_replica.cc:330] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: stopping tablet replica
I20260620 07:14:12.424324 17336 raft_consensus.cc:2229] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:12.428948 17335 raft_consensus.cc:2229] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:12.429479 17335 raft_consensus.cc:2258] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:12.430752 17336 raft_consensus.cc:2258] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:12.431180 17335 ts_tablet_manager.cc:1905] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:12.434001 17335 ts_tablet_manager.cc:1918] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:12.440280 17335 log.cc:1196] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/9a0d37c3cda741c89031f584b463556c
I20260620 07:14:12.440717 17335 ts_tablet_manager.cc:1939] T 9a0d37c3cda741c89031f584b463556c P 42440624f787475b8f6662c693f47a31: Deleting consensus metadata
I20260620 07:14:12.440961 17336 ts_tablet_manager.cc:1905] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:12.442081 16796 catalog_manager.cc:4841] TS 42440624f787475b8f6662c693f47a31 (127.3.245.2:43373): tablet 9a0d37c3cda741c89031f584b463556c (table test-table [id=ec8c5d25ec8d459abecf58c10d2719c7]) successfully deleted
W20260620 07:14:12.443043 16797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:34449 (error 108)
I20260620 07:14:12.443130 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17013
I20260620 07:14:12.445053 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:12.444467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34449 (local address 127.0.0.1:47934)
0620 07:14:12.444899 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:12.444968 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351}
W20260620 07:14:12.445256 16797 catalog_manager.cc:4584] TS fbd77b4497d84cc08b0ba11e02095c8d (127.3.245.1:34449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a0d37c3cda741c89031f584b463556c: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34449: connect: Connection refused (error 111)
I20260620 07:14:12.446254 17336 ts_tablet_manager.cc:1918] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:12.446395 17336 log.cc:1196] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/9a0d37c3cda741c89031f584b463556c
I20260620 07:14:12.446770 17336 ts_tablet_manager.cc:1939] T 9a0d37c3cda741c89031f584b463556c P 771874bec91e45289ea187e0ffefbb9d: Deleting consensus metadata
I20260620 07:14:12.448271 16798 catalog_manager.cc:4841] TS 771874bec91e45289ea187e0ffefbb9d (127.3.245.3:41007): tablet 9a0d37c3cda741c89031f584b463556c (table test-table [id=ec8c5d25ec8d459abecf58c10d2719c7]) successfully deleted
I20260620 07:14:12.452718 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17147
I20260620 07:14:12.462513 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 16778
I20260620 07:14:12.469692 4052 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:14:12.814851 4052 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:14:12.814967 4052 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:14:13.160236 4052 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:14:13.160 UTC [15845] LOG: received smart shutdown request
2026-06-20 07:14:13.164 UTC [15845] LOG: background worker "logical replication launcher" (PID 15854) exited with exit code 1
2026-06-20 07:14:13.164 UTC [15849] LOG: shutting down
2026-06-20 07:14:13.329 UTC [15845] LOG: database system is shut down
2026-06-20T07:14:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49378 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:13 dist-test-slave-ldfr krb5kdc[17347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:13 dist-test-slave-ldfr krb5kdc[17347](info): set up 2 sockets
Jun 20 07:14:13 dist-test-slave-ldfr krb5kdc[17347](info): commencing operation
krb5kdc: starting...
W20260620 07:14:14.500806 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:14:14 dist-test-slave-ldfr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:14:14.623983 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36997
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:14.733752 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:14.734055 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:14.734098 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:14.738034 17360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:14.738095 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:14.738116 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:14.738132 17360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:14.738147 17360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:14.743274 17360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:14.744279 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:14.745267 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:14.753834 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:14.753803 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:14.753953 17360 server_base.cc:1034] running on GCE node
W20260620 07:14:14.753858 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:14.754653 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:14.755820 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:14.756989 17360 hybrid_clock.cc:648] HybridClock initialized: now 1781939654756970 us; error 37 us; skew 500 ppm
Jun 20 07:14:14 dist-test-slave-ldfr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:14.759961 17360 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:14.760987 17360 webserver.cc:460] Webserver started at http://127.3.245.62:34123/ using document root <none> and password file <none>
I20260620 07:14:14.761564 17360 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:14.761628 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:14.761840 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:14.763922 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "fbae12f8dad246559b575659aa13db8d"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "cf70e62158e5d009f90c6cfd130556e4"
server_key_iv: "85c0a0fc5fca6a5156cf9cfd919734c0"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.764487 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "fbae12f8dad246559b575659aa13db8d"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "cf70e62158e5d009f90c6cfd130556e4"
server_key_iv: "85c0a0fc5fca6a5156cf9cfd919734c0"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.768328 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:14:14.771241 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:14.772548 17360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:14.772754 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "fbae12f8dad246559b575659aa13db8d"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "cf70e62158e5d009f90c6cfd130556e4"
server_key_iv: "85c0a0fc5fca6a5156cf9cfd919734c0"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.772853 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:14.806439 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:14.809653 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:14.809863 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:14.817157 17360 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:36997
I20260620 07:14:14.817176 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:36997 every 8 connection(s)
I20260620 07:14:14.818123 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:14.820850 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17360
I20260620 07:14:14.821013 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:14:14.821082 17428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:14.821365 4052 external_mini_cluster.cc:1428] Setting key e55acc0b72cffa23d32646d7392f7cce
I20260620 07:14:14.825560 17428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.245.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.61@KRBTEST.COM" created.
Jun 20 07:14:14 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:14.841746 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.827894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.3.245.62:36997)
0620 07:14:14.828400 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:14.828422 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:14.828712 (+ 290us) server_negotiation.cc:407] Connection header received
0620 07:14:14.829772 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:14.829776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:14.830032 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:14.830188 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:14.831115 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.832200 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.833600 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.833971 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.834141 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:14.836873 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:14.836879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:14.836889 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:14.836910 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:14.838513 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.838991 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.838995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.838996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.839093 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:14.839419 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:14.839422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:14.839423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:14.839794 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:14.839968 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:14.840355 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 07:14:14.840598 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":151,"threads_started":1}
I20260620 07:14:14.841746 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.827894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36997 (local address 127.0.0.1:43750)
0620 07:14:14.828536 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.828584 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:14.829488 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:14.830242 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:14.830246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:14.830583 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:14.830955 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:14.830972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.832372 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.832379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:14.833479 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:14.833488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:14.834126 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:14.834129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:14.834237 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:14.834840 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:14.834858 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:14.836697 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:14.838629 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:14.838640 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:14.838650 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:14.838870 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:14.839192 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:14.839195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:14.839197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:14.839310 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:14.839947 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:14.839954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:14.840097 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 07:14:14.840260 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 07:14:14.840694 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":108,"spinlock_wait_cycles":47872,"thread_start_us":156,"threads_started":1}
I20260620 07:14:14.844429 17428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:14.845285 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.844993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45694)
0620 07:14:14.845108 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.845177 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:14:14.845572 17377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
I20260620 07:14:14.846460 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.846258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45706)
0620 07:14:14.846351 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.846408 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 07:14:14.846645 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43983: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
I20260620 07:14:14.847117 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43983
--webserver_interface=127.3.245.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:14.895948 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } attempt: 1
I20260620 07:14:14.897064 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.896667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45718)
0620 07:14:14.896876 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.896940 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:14:14.897873 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.897643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45720)
0620 07:14:14.897759 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.897812 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:14:14.898015 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43983: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
W20260620 07:14:14.959985 17435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:14.960309 17435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:14.960387 17435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:14.964571 17435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:14.964711 17435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:14.964764 17435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:14.964823 17435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:14.964864 17435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:14.966203 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } attempt: 2
I20260620 07:14:14.967056 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.966731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45732)
0620 07:14:14.966893 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.966974 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 07:14:14.967845 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:14.967613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45744)
0620 07:14:14.967733 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:14.967789 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:14:14.967995 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43983: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43983: connect: Connection refused (error 111)
I20260620 07:14:14.970646 17435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.245.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:14.971750 17435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:14.972892 17435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:14.981344 17441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:14.981303 17440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:14.981303 17443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:14.981797 17435 server_base.cc:1034] running on GCE node
I20260620 07:14:14.982285 17435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:14.983342 17435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:14.984539 17435 hybrid_clock.cc:648] HybridClock initialized: now 1781939654984526 us; error 35 us; skew 500 ppm
Jun 20 07:14:14 dist-test-slave-ldfr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:14.987290 17435 init.cc:377] Logged in from keytab as kudu/127.3.245.61@KRBTEST.COM (short username kudu)
I20260620 07:14:14.988190 17435 webserver.cc:460] Webserver started at http://127.3.245.61:45367/ using document root <none> and password file <none>
I20260620 07:14:14.988814 17435 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:14.988902 17435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:14.989130 17435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:14.991142 17435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data/instance:
uuid: "ee28486b60be4fb2a04c2faccbb9c844"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "edb270390b4c9e352e0be3cff6c60656"
server_key_iv: "8db3d0aabcd7c0b59d68bc23322042fe"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.991735 17435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal/instance:
uuid: "ee28486b60be4fb2a04c2faccbb9c844"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "edb270390b4c9e352e0be3cff6c60656"
server_key_iv: "8db3d0aabcd7c0b59d68bc23322042fe"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.995703 17435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:14:14.998283 17450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:14.999670 17435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:14.999842 17435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "ee28486b60be4fb2a04c2faccbb9c844"
format_stamp: "Formatted at 2026-06-20 07:14:14 on dist-test-slave-ldfr"
server_key: "edb270390b4c9e352e0be3cff6c60656"
server_key_iv: "8db3d0aabcd7c0b59d68bc23322042fe"
server_key_version: "encryptionkey@0"
I20260620 07:14:14.999979 17435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:15.019131 17435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:15.022413 17435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:15.022622 17435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:15.029917 17435 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.61:43983
I20260620 07:14:15.029937 17502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.61:43983 every 8 connection(s)
I20260620 07:14:15.030866 17435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
I20260620 07:14:15.033607 17503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:15.034192 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17435
I20260620 07:14:15.034360 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-1/wal/instance
I20260620 07:14:15.034693 4052 external_mini_cluster.cc:1428] Setting key c7985a132166b41f0421c9e5dcec2c7c
I20260620 07:14:15.038250 17503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.245.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.60@KRBTEST.COM" created.
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:15.054386 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.040197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.3.245.62:36997)
0620 07:14:15.040376 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.040383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.041272 (+ 889us) server_negotiation.cc:407] Connection header received
0620 07:14:15.042234 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.042238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.042300 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.042418 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.043724 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.044249 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.046006 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.046239 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.046326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.049062 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.049067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.049070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.049102 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.051752 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.052424 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.052429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.052432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.052541 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.053020 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.053024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.053026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.053293 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.053431 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.054043 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.054218 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:14:15.055279 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.040551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36997 (local address 127.0.0.1:43764)
0620 07:14:15.041095 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.041148 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.041993 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.042597 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.042605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.042931 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.043574 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.043589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.044412 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.044415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.045838 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.045864 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.046406 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.046413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.046673 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.047316 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.047332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.048873 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.051959 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.051964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.051980 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.052285 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.052697 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.052699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.052700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.052845 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.053430 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.053436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.053700 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.053921 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.054309 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
I20260620 07:14:15.057230 17503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:15.059186 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
I20260620 07:14:15.063691 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:42625
--webserver_interface=127.3.245.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:14:15.069525 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.057877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45752)
0620 07:14:15.058043 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.058056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.058125 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.058760 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.058763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.058914 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.059019 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.059024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.060351 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.060354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.061222 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.061232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.061943 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.061947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.062044 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.062624 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.062643 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.064922 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.067739 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.067744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.067746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.067984 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.068338 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.068349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.068495 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.069096 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.069099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.069178 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.069252 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.069357 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 07:14:15.070065 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.059701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.3.245.61:43983)
0620 07:14:15.059997 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.060006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.060016 (+ 10us) server_negotiation.cc:407] Connection header received
0620 07:14:15.060041 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.060043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.060074 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.060162 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.060680 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.061388 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.062339 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.062657 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.062807 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.065068 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.065083 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.065104 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.065207 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.067797 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068329 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.068332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.068334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.068453 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068820 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.068823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.068825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.068977 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.069117 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.069590 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.069885 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"spinlock_wait_cycles":42752,"thread_start_us":116,"threads_started":1}
I20260620 07:14:15.070091 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.058006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.3.245.61:43983)
0620 07:14:15.058378 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.058393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.058419 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:14:15.058473 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.058479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.058644 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.058759 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.059157 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.060242 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.061339 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.061834 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.061917 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.065100 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.065106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.065111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.065146 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.067614 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068109 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.068117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.068121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.068223 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068587 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.068589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.068591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.068977 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.069716 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.069782 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.069943 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":272,"thread_start_us":188,"threads_started":1}
I20260620 07:14:15.070842 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.059648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45758)
0620 07:14:15.059767 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.059783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.059883 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.060183 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.060186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.060396 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.060517 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.060524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.061486 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.061488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.062208 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.062223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.062813 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.062818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.062931 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.063450 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.063476 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.064864 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.067917 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.067923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.067925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.068214 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.068565 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.068570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.068575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.068709 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.069100 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.069105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.069317 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.069462 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.069927 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 07:14:15.071890 17428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:15.072136 17503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:15.072695 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.072379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43152)
0620 07:14:15.072496 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.072549 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:14:15.073128 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.072676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43158)
0620 07:14:15.072943 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.073046 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260620 07:14:15.073354 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.073124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43174)
0620 07:14:15.073254 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.073306 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 07:14:15.073464 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
W20260620 07:14:15.073450 17453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
I20260620 07:14:15.074298 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.074108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43190)
0620 07:14:15.074210 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.074251 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:14:15.074461 17503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
I20260620 07:14:15.104689 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 1
I20260620 07:14:15.105695 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.105334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43198)
0620 07:14:15.105532 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.105605 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 07:14:15.106639 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.106384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43200)
0620 07:14:15.106502 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.106571 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:14:15.106822 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
I20260620 07:14:15.123693 17503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 1
I20260620 07:14:15.124811 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.124346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43202)
0620 07:14:15.124596 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.124728 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 07:14:15.125571 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.125358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43204)
0620 07:14:15.125464 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.125516 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 07:14:15.125701 17503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
I20260620 07:14:15.182019 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 2
I20260620 07:14:15.182866 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.182554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43214)
0620 07:14:15.182706 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.182779 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:14:15.183640 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.183391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43222)
0620 07:14:15.183502 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.183581 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:14:15.183799 17428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
W20260620 07:14:15.186327 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:15.186689 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:15.186789 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:15.191077 17510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:15.191166 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:15.191215 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:15.191272 17510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:15.191309 17510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:15.193863 17503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 2
I20260620 07:14:15.194712 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.194383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43238)
0620 07:14:15.194576 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.194633 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:14:15.195503 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.195244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43240)
0620 07:14:15.195353 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.195424 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:14:15.195662 17503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:42625: Network error: Client connection negotiation failed: client connection to 127.3.245.60:42625: connect: Connection refused (error 111)
I20260620 07:14:15.197480 17510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36997,127.3.245.61:43983,127.3.245.60:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.245.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:15.198642 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:15.199680 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:15.209129 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.209129 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:15.209175 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:15.209288 17510 server_base.cc:1034] running on GCE node
I20260620 07:14:15.209981 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:15.211145 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:15.212345 17510 hybrid_clock.cc:648] HybridClock initialized: now 1781939655212310 us; error 54 us; skew 500 ppm
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:15.215292 17510 init.cc:377] Logged in from keytab as kudu/127.3.245.60@KRBTEST.COM (short username kudu)
I20260620 07:14:15.216284 17510 webserver.cc:460] Webserver started at http://127.3.245.60:35473/ using document root <none> and password file <none>
I20260620 07:14:15.216938 17510 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:15.217036 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:15.217278 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:15.219471 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data/instance:
uuid: "3b383c99178940dbb9730eb6d9348b26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-ldfr"
server_key: "b331d8e029468d52c5711d7c597a7187"
server_key_iv: "e0f9608ab48e2e632a03db05d9784905"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.220062 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal/instance:
uuid: "3b383c99178940dbb9730eb6d9348b26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-ldfr"
server_key: "b331d8e029468d52c5711d7c597a7187"
server_key_iv: "e0f9608ab48e2e632a03db05d9784905"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.224035 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:14:15.226676 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:15.227962 17510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:15.228127 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "3b383c99178940dbb9730eb6d9348b26"
format_stamp: "Formatted at 2026-06-20 07:14:15 on dist-test-slave-ldfr"
server_key: "b331d8e029468d52c5711d7c597a7187"
server_key_iv: "e0f9608ab48e2e632a03db05d9784905"
server_key_version: "encryptionkey@0"
I20260620 07:14:15.228247 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:15.242683 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:15.246330 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:15.246565 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:15.254364 17510 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.60:42625
I20260620 07:14:15.254410 17578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.60:42625 every 8 connection(s)
I20260620 07:14:15.255414 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
I20260620 07:14:15.258559 17579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:15.261615 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17510
I20260620 07:14:15.261765 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781939488540940-4052-0/minicluster-data/master-2/wal/instance
I20260620 07:14:15.262055 4052 external_mini_cluster.cc:1428] Setting key 991bf2ca036ca778ef5b375673505bad
I20260620 07:14:15.262965 17579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:14:15.276173 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.265010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.3.245.62:36997)
0620 07:14:15.265142 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.265146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.265467 (+ 321us) server_negotiation.cc:407] Connection header received
0620 07:14:15.265655 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.265658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.265716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.265791 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.266589 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.267116 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.268122 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.268360 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.268481 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.271282 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.271286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.271288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.271316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.273254 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.274104 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.274110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.274112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.274240 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.274628 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.274633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.274635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.274933 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.275060 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.275812 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.275961 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:14:15.277380 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.264906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36997 (local address 127.0.0.1:43768)
0620 07:14:15.265301 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.265342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.265500 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.265838 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.265843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.266194 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.266443 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.266457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.267229 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.267234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.267996 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.268005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.268521 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.268525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.268751 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.269371 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.269396 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.271006 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.273398 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.273408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.273420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.273889 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.274354 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.274356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.274358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.274516 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.275132 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.275141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.275501 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.275750 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.276253 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":67,"thread_start_us":150,"threads_started":1}
I20260620 07:14:15.277400 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.263491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.3.245.60:42625)
0620 07:14:15.264027 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.264044 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.264083 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:14:15.264994 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.265022 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.265461 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.265736 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.266586 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.267425 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.268300 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.268725 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.268840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.271667 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.271673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.271686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.271707 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.273885 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.274702 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.274707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.274708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.274776 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.275153 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.275156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.275157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.275515 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.275712 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.275990 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.276562 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":357,"server-negotiator.queue_time_us":335,"thread_start_us":141,"threads_started":1}
I20260620 07:14:15.278807 17579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:15.283020 17428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 3
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:15.286811 17503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } attempt: 3
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:14:15.289351 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.279340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43983 (local address 127.0.0.1:45760)
0620 07:14:15.279709 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.279726 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.279814 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.280255 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.280261 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.280464 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.280564 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.280570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.281495 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.281499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.282137 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.282146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.282729 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.282732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.282828 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.283590 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.283611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.285282 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.287578 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.287582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.287584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.287823 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.288190 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.288196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.288201 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.288293 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.288816 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.288819 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.288917 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.289023 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.289171 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260620 07:14:15.289870 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.279428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.3.245.61:43983)
0620 07:14:15.279564 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.279568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.279792 (+ 224us) server_negotiation.cc:407] Connection header received
0620 07:14:15.279963 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.279967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.280025 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.280082 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.280720 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.281383 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.282248 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.282621 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.282700 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.285420 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.285424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.285427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.285462 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.287435 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.287953 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.287958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.287960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.288066 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.288400 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.288404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.288405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.288697 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.289080 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.289148 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.289257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:14:15.290550 17579 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:14:15.291029 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.279909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.3.245.62:36997)
0620 07:14:15.280021 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.280025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.280067 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:14:15.280178 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.280184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.280240 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.280315 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.281092 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.281687 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.282546 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.282818 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.282923 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.286172 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.286178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.286184 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.286220 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.288137 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.288970 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.288975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.288980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.289073 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.289395 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.289398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.289400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.289637 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.289842 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.290521 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.290643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:14:15.294749 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.283471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43260)
0620 07:14:15.283764 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.283784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.283882 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.284102 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.284105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.284298 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.284424 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.284432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.285536 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.285542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.286608 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.286622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.287601 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.287605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.287715 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.288140 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.288155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.289637 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.292451 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.292454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.292457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.292944 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.293633 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.293637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.293639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.293734 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.294207 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.294211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.294307 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.294397 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.294558 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"spinlock_wait_cycles":21504}
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:14:15.295439 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.283533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.3.245.60:42625)
0620 07:14:15.283679 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.283685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.283844 (+ 159us) server_negotiation.cc:407] Connection header received
0620 07:14:15.283956 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.283959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.283998 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.284088 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.284558 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.285389 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.286764 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.287466 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.287930 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.289774 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.289778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.289780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.289804 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.292317 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.293418 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.293421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.293423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.293517 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.293852 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.293858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.293860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.294087 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.295064 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.295144 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.295272 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939655, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:14:15.299242 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: Bootstrap starting.
I20260620 07:14:15.299647 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.287809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43274)
0620 07:14:15.288179 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.288196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.288288 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.288707 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.288710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.288892 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.288989 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.288997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.291772 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.291777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.292656 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.292669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.293323 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.293327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.293418 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.293897 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.293914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.295443 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.298019 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.298025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.298030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.298267 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.298598 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.298601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.298603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.298692 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.299171 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.299174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.299270 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.299357 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.299500 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260620 07:14:15.300333 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.287563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.3.245.60:42625)
0620 07:14:15.288414 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.288423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.288436 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:14:15.288469 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.288472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.288551 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.290263 (+ 1712us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.290368 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.291637 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.292803 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.293198 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.293630 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.295605 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.295609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.295612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.295636 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.297889 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.298393 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.298396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.298398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.298480 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.298799 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.298802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.298804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.299054 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.299942 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.300019 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.300136 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":101,"threads_started":1}
Jun 20 07:14:15 dist-test-slave-ldfr krb5kdc[17347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:14:15.302150 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.291062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:42625 (local address 127.0.0.1:43284)
0620 07:14:15.291437 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:15.291450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:15.291517 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:15.292111 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:15.292117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:15.292323 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:15.292424 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.292433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.293434 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.293437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.294265 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:15.294277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.295073 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.295077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:15.295212 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.295743 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:15.295758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:15.297756 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:15.300194 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.300197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.300199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.300763 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.301049 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:15.301052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:15.301054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:15.301133 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:15.301606 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:15.301609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:15.301703 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 07:14:15.301808 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 07:14:15.301951 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260620 07:14:15.302297 17428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:15.302924 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.291232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.3.245.60:42625)
0620 07:14:15.291535 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.291552 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.291567 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:14:15.291604 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.291608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.291879 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.291968 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.292546 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.293310 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.294385 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.294729 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.295420 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.297902 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.297906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.297908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.297935 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.300070 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.300872 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.300875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.300876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.300961 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.301230 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.301235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.301238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.301472 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.301569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.302609 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.302729 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":112,"threads_started":1}
I20260620 07:14:15.303480 17503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: Bootstrap starting.
I20260620 07:14:15.303480 17428 log.cc:824] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:15.304685 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.292775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.3.245.61:43983)
0620 07:14:15.293101 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.293105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.293117 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:14:15.293151 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.293154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.293205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.293285 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.294377 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.295103 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.296682 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.297300 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.297392 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.301039 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.301043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.301046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.301069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.302751 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.303287 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.303293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.303298 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.303380 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.303724 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.303727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.303729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.303968 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.304170 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.304362 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.304488 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260620 07:14:15.306319 17503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:15.307155 17503 log.cc:824] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:15.307684 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: Bootstrap starting.
I20260620 07:14:15.308086 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:15.309401 17503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: No bootstrap required, opened a new log
I20260620 07:14:15.309540 17428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: No bootstrap required, opened a new log
I20260620 07:14:15.310333 17579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:15.311165 17579 log.cc:824] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:15.313476 17503 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.313694 17503 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:15.313752 17503 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee28486b60be4fb2a04c2faccbb9c844, State: Initialized, Role: FOLLOWER
I20260620 07:14:15.313791 17428 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.313998 17428 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:15.314049 17428 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbae12f8dad246559b575659aa13db8d, State: Initialized, Role: FOLLOWER
I20260620 07:14:15.314162 17579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: No bootstrap required, opened a new log
I20260620 07:14:15.314532 17428 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.314530 17503 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.315927 17428 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:15.316468 17503 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:15.315387 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.316972 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.319201 17579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.319506 17579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:15.319577 17579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b383c99178940dbb9730eb6d9348b26, State: Initialized, Role: FOLLOWER
I20260620 07:14:15.319746 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.319893 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.320150 17579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.321738 17579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:14:15.324136 17614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:15.324221 17614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:15.321803 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.310421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.3.245.62:36997)
0620 07:14:15.310544 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.310549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.311037 (+ 488us) server_negotiation.cc:407] Connection header received
0620 07:14:15.311213 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.311216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.311266 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.311316 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.312323 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.313146 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.314286 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.314611 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.314716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.316538 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.316544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.316548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.316580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.318498 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.319101 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.319106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.319108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.319198 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.319544 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.319548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.319550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.319802 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.319933 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.321520 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.321654 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:14:15.330890 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.316976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.3.245.61:43983)
0620 07:14:15.317103 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.317108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.320899 (+ 3791us) server_negotiation.cc:407] Connection header received
0620 07:14:15.320955 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.320958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.321008 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.321079 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.322669 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.323434 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.324516 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.324832 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.324934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.326683 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.326687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.326690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.326717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.328454 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.329263 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.329267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.329269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.329357 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.329666 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.329672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.329677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.329892 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.330029 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.330612 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.330730 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:14:15.332084 17360 master_runner.cc:263] No leader master found from master fbae12f8dad246559b575659aa13db8d
I20260620 07:14:15.332161 17360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:14:15.326892 17435 master_runner.cc:186] Error getting master registration for 127.3.245.60:42625: OK, instance_id { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" instance_seqno: 1781939655252112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:14:15.332469 17435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:14:15.336386 17510 master_runner.cc:263] No leader master found from master 3b383c99178940dbb9730eb6d9348b26
I20260620 07:14:15.336481 17510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:14:15.338387 17619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:15.338495 17619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:15.339061 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:15.317545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.3.245.60:42625)
0620 07:14:15.317673 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:15.317677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:15.321186 (+ 3509us) server_negotiation.cc:407] Connection header received
0620 07:14:15.321235 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:15.321239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:15.321288 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:15.321358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:15.325268 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.326016 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.327076 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:15.327437 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:15.327511 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:15.335110 (+ 7599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:15.335114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:15.335117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:15.335149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:15.337010 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.337534 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.337538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.337540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.337640 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:15.337957 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:15.337961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:15.337962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:15.338214 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:15.338348 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:15.338751 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 07:14:15.338884 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:14:15.340794 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.340936 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:14:15.341931 17632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:15.342023 17632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:15.406977 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:15.407197 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.408347 17594 leader_election.cc:288] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee28486b60be4fb2a04c2faccbb9c844 (127.3.245.61:43983), 3b383c99178940dbb9730eb6d9348b26 (127.3.245.60:42625)
I20260620 07:14:15.409216 17553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbae12f8dad246559b575659aa13db8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b383c99178940dbb9730eb6d9348b26" is_pre_election: true
I20260620 07:14:15.409581 17553 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbae12f8dad246559b575659aa13db8d in term 0.
I20260620 07:14:15.410140 17378 leader_election.cc:302] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b383c99178940dbb9730eb6d9348b26, fbae12f8dad246559b575659aa13db8d; no voters:
I20260620 07:14:15.411026 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbae12f8dad246559b575659aa13db8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee28486b60be4fb2a04c2faccbb9c844" is_pre_election: true
I20260620 07:14:15.411374 17476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbae12f8dad246559b575659aa13db8d in term 0.
I20260620 07:14:15.412065 17594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:15.412168 17594 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:15.412220 17594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:15.413625 17594 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.414023 17594 leader_election.cc:288] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [CANDIDATE]: Term 1 election: Requested vote from peers ee28486b60be4fb2a04c2faccbb9c844 (127.3.245.61:43983), 3b383c99178940dbb9730eb6d9348b26 (127.3.245.60:42625)
I20260620 07:14:15.414515 17553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbae12f8dad246559b575659aa13db8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b383c99178940dbb9730eb6d9348b26"
I20260620 07:14:15.414659 17553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:15.416025 17553 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbae12f8dad246559b575659aa13db8d in term 1.
I20260620 07:14:15.416430 17378 leader_election.cc:302] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b383c99178940dbb9730eb6d9348b26, fbae12f8dad246559b575659aa13db8d; no voters:
I20260620 07:14:15.416978 17476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbae12f8dad246559b575659aa13db8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee28486b60be4fb2a04c2faccbb9c844"
I20260620 07:14:15.417129 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:15.418442 17476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbae12f8dad246559b575659aa13db8d in term 1.
I20260620 07:14:15.421581 17594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:15.421850 17594 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [term 1 LEADER]: Becoming Leader. State: Replica: fbae12f8dad246559b575659aa13db8d, State: Running, Role: LEADER
I20260620 07:14:15.422231 17594 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } }
I20260620 07:14:15.424098 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbae12f8dad246559b575659aa13db8d. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.424212 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:15.424649 17637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:15.425616 17637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:15.434559 17476 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [term 1 FOLLOWER]: Refusing update from remote peer fbae12f8dad246559b575659aa13db8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:15.435891 17553 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [term 1 FOLLOWER]: Refusing update from remote peer fbae12f8dad246559b575659aa13db8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:15.438143 17594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:15.438683 17594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:15.443111 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbae12f8dad246559b575659aa13db8d. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.443243 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.447126 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbae12f8dad246559b575659aa13db8d. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.447288 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.456681 17635 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.456848 17635 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:15.457783 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.457914 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.458271 17638 mvcc.cc:205] Tried to move back new op lower bound from 7298824828647178240 to 7298824828612820992. Current Snapshot: MvccSnapshot[applied={T|T < 7298824828647178240}]
I20260620 07:14:15.462814 17620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.462983 17620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:14:15.463274 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fbae12f8dad246559b575659aa13db8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbae12f8dad246559b575659aa13db8d" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36997 } } peers { permanent_uuid: "ee28486b60be4fb2a04c2faccbb9c844" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43983 } } peers { permanent_uuid: "3b383c99178940dbb9730eb6d9348b26" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 42625 } } }
I20260620 07:14:15.463413 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:15.463744 17641 mvcc.cc:205] Tried to move back new op lower bound from 7298824828647178240 to 7298824828612820992. Current Snapshot: MvccSnapshot[applied={T|T < 7298824828647178240}]
I20260620 07:14:15.467772 17639 mvcc.cc:205] Tried to move back new op lower bound from 7298824828647178240 to 7298824828612820992. Current Snapshot: MvccSnapshot[applied={T|T < 7298824828647178240}]
I20260620 07:14:15.473701 17637 catalog_manager.cc:1347] Generated new cluster ID: ccb12ce12ee74f569423b8cec3a05e79
I20260620 07:14:15.473805 17637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:15.533187 17637 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:15.534292 17637 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:15.542517 17637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbae12f8dad246559b575659aa13db8d: Generated new TSK 0
I20260620 07:14:15.543787 17637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:16.340010 17619 catalog_manager.cc:1259] Loaded cluster ID: ccb12ce12ee74f569423b8cec3a05e79
I20260620 07:14:16.340104 17619 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: loading cluster ID for follower catalog manager: success
I20260620 07:14:16.341902 17619 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: acquiring CA information for follower catalog manager: success
I20260620 07:14:16.343173 17619 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee28486b60be4fb2a04c2faccbb9c844: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:14:16.343413 17632 catalog_manager.cc:1259] Loaded cluster ID: ccb12ce12ee74f569423b8cec3a05e79
I20260620 07:14:16.343498 17632 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: loading cluster ID for follower catalog manager: success
I20260620 07:14:16.345264 17632 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: acquiring CA information for follower catalog manager: success
I20260620 07:14:16.346535 17632 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b383c99178940dbb9730eb6d9348b26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:14:16.605643 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17360
I20260620 07:14:16.612721 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17435
I20260620 07:14:16.618901 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17510
2026-06-20T07:14:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3173 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:16 dist-test-slave-ldfr krb5kdc[17652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:16 dist-test-slave-ldfr krb5kdc[17652](info): set up 2 sockets
Jun 20 07:14:16 dist-test-slave-ldfr krb5kdc[17652](info): commencing operation
krb5kdc: starting...
W20260620 07:14:17.717134 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:14:17 dist-test-slave-ldfr krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:14:17.843803 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45779
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:17.954739 17665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:17.955088 17665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:17.955178 17665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:17.959373 17665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:17.959476 17665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:17.959515 17665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:17.959549 17665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:17.959581 17665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:17.964922 17665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:17.965994 17665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:17.967015 17665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:17.975762 17673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:17.975790 17670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:17.975762 17671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:17.976156 17665 server_base.cc:1034] running on GCE node
I20260620 07:14:17.976773 17665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:17.977892 17665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:17.979061 17665 hybrid_clock.cc:648] HybridClock initialized: now 1781939657979039 us; error 35 us; skew 500 ppm
Jun 20 07:14:17 dist-test-slave-ldfr krb5kdc[17652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939657, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:17.981827 17665 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:17.982714 17665 webserver.cc:460] Webserver started at http://127.3.245.62:34525/ using document root <none> and password file <none>
I20260620 07:14:17.983258 17665 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:17.983345 17665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:17.983595 17665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:17.985631 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "1134f0469c044b6fa96599cb4d5a9919"
format_stamp: "Formatted at 2026-06-20 07:14:17 on dist-test-slave-ldfr"
server_key: "b0c73e3ef57ae666df52058cf7efabb6"
server_key_iv: "228c731637a630bc9d380ea34dbbb63f"
server_key_version: "encryptionkey@0"
I20260620 07:14:17.986174 17665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "1134f0469c044b6fa96599cb4d5a9919"
format_stamp: "Formatted at 2026-06-20 07:14:17 on dist-test-slave-ldfr"
server_key: "b0c73e3ef57ae666df52058cf7efabb6"
server_key_iv: "228c731637a630bc9d380ea34dbbb63f"
server_key_version: "encryptionkey@0"
I20260620 07:14:17.989809 17665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:14:17.992065 17680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:17.993250 17665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:14:17.993387 17665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "1134f0469c044b6fa96599cb4d5a9919"
format_stamp: "Formatted at 2026-06-20 07:14:17 on dist-test-slave-ldfr"
server_key: "b0c73e3ef57ae666df52058cf7efabb6"
server_key_iv: "228c731637a630bc9d380ea34dbbb63f"
server_key_version: "encryptionkey@0"
I20260620 07:14:17.993518 17665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:18.015136 17665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:18.018414 17665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:18.018654 17665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:18.026414 17665 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45779
I20260620 07:14:18.026662 17732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45779 every 8 connection(s)
I20260620 07:14:18.027485 17665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:18.030297 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17665
I20260620 07:14:18.030332 17733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:18.030458 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:14:18.030747 4052 external_mini_cluster.cc:1428] Setting key 9aed1414df50cc4cf5782fa6ddc5819c
I20260620 07:14:18.037537 17733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919: Bootstrap starting.
I20260620 07:14:18.039976 17733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:18.040871 17733 log.cc:824] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:18 dist-test-slave-ldfr krb5kdc[17652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:18.042810 17733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919: No bootstrap required, opened a new log
I20260620 07:14:18.046386 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:18.032094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.3.245.62:45779)
0620 07:14:18.032503 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:18.032524 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:18.032564 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:14:18.033730 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:18.033755 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:18.034184 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:18.034551 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:18.035305 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:18.036900 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:18.037913 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:18.038585 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:18.038760 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:18.041642 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:18.041647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:18.041660 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:18.041689 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:18.043518 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:18.043967 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:18.043978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:18.043982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:18.044089 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:18.044388 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:18.044393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:18.044394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:18.044878 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:18.045093 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:18.045369 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:14:18.045656 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260620 07:14:18.046437 17733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } }
I20260620 07:14:18.046758 17733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:18.046824 17733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1134f0469c044b6fa96599cb4d5a9919, State: Initialized, Role: FOLLOWER
I20260620 07:14:18.047278 17733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } }
I20260620 07:14:18.047426 17733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:18.047502 17733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:18.047621 17733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:18.048703 17733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } }
I20260620 07:14:18.049057 17733 leader_election.cc:302] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1134f0469c044b6fa96599cb4d5a9919; no voters:
I20260620 07:14:18.049340 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:14:18.049398 17733 leader_election.cc:288] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:18.049510 17738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:18.049690 17738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [term 1 LEADER]: Becoming Leader. State: Replica: 1134f0469c044b6fa96599cb4d5a9919, State: Running, Role: LEADER
I20260620 07:14:18.050036 17738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } }
I20260620 07:14:18.050710 17733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:18.051810 17739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1134f0469c044b6fa96599cb4d5a9919. Latest consensus state: current_term: 1 leader_uuid: "1134f0469c044b6fa96599cb4d5a9919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } } }
I20260620 07:14:18.051944 17739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:18.052282 17740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1134f0469c044b6fa96599cb4d5a9919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1134f0469c044b6fa96599cb4d5a9919" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45779 } } }
I20260620 07:14:18.052453 17740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919 [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:18.052345 17747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:18.055671 17747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:18.061960 17747 catalog_manager.cc:1347] Generated new cluster ID: ee3de90a064240138be6a1ee80dde84d
I20260620 07:14:18.062055 17747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:18.089695 17747 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:18.091105 17747 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:18.102469 17747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1134f0469c044b6fa96599cb4d5a9919: Generated new TSK 0
I20260620 07:14:18.103196 17747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:14:18.156733 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17665
2026-06-20T07:14:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1534 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:18 dist-test-slave-ldfr krb5kdc[17760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:18 dist-test-slave-ldfr krb5kdc[17760](info): set up 2 sockets
Jun 20 07:14:18 dist-test-slave-ldfr krb5kdc[17760](info): commencing operation
krb5kdc: starting...
W20260620 07:14:19.225122 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:14:19 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:14:19.357234 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:19.476022 17773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:19.476377 17773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:19.476467 17773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:19.480834 17773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:19.480954 17773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:19.480994 17773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:19.481033 17773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:19.481070 17773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:19.486845 17773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:19.488008 17773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:19.489079 17773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:19.498277 17779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:19.498464 17773 server_base.cc:1034] running on GCE node
W20260620 07:14:19.498278 17778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.498730 17781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:19.499351 17773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:19.500835 17773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:19.502230 17773 hybrid_clock.cc:648] HybridClock initialized: now 1781939659502193 us; error 52 us; skew 500 ppm
Jun 20 07:14:19 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:19.505533 17773 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:19.506561 17773 webserver.cc:460] Webserver started at http://127.3.245.62:36723/ using document root <none> and password file <none>
I20260620 07:14:19.507175 17773 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:19.507251 17773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:19.507474 17773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:19.509770 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "0bc7ff63f69d4b518aa8c240edd3064f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "6f059a5658f5dcc23918fb59d4d668bb"
server_key_iv: "79a87024134cd50e995304b74059223d"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.510383 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "0bc7ff63f69d4b518aa8c240edd3064f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "6f059a5658f5dcc23918fb59d4d668bb"
server_key_iv: "79a87024134cd50e995304b74059223d"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.514621 17773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:19.517470 17788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.519006 17773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:14:19.519141 17773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "0bc7ff63f69d4b518aa8c240edd3064f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "6f059a5658f5dcc23918fb59d4d668bb"
server_key_iv: "79a87024134cd50e995304b74059223d"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.519322 17773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:19.536839 17773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:19.540549 17773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:19.540827 17773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:19.549108 17773 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45645
I20260620 07:14:19.549566 17840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45645 every 8 connection(s)
I20260620 07:14:19.550180 17773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:19.553287 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:19.554909 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17773
I20260620 07:14:19.555027 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:14:19.555382 4052 external_mini_cluster.cc:1428] Setting key 452fb07c72dff6e81332d173fefc4291
I20260620 07:14:19.560413 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap starting.
I20260620 07:14:19.563519 17841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:19.564549 17841 log.cc:824] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:19 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:19.566743 17841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: No bootstrap required, opened a new log
I20260620 07:14:19.570680 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:19.556942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.3.245.62:45645)
0620 07:14:19.557372 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:19.557389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:19.557424 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:14:19.558065 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:19.558081 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:19.558419 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:19.558692 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:19.559702 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.560807 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.562147 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.562636 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.562761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:19.565975 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:19.565981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:19.565995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:19.566022 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:19.567776 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.568247 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.568252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.568257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.568374 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.568720 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.568724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.568725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.569136 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:19.569377 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:19.569603 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 07:14:19.569910 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":141,"threads_started":1}
I20260620 07:14:19.570816 17841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:19.571118 17841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:19.571202 17841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Initialized, Role: FOLLOWER
I20260620 07:14:19.571682 17841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:19.571843 17841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:19.571921 17841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:19.572050 17841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:19.573278 17841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:19.573634 17841 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc7ff63f69d4b518aa8c240edd3064f; no voters:
I20260620 07:14:19.573985 17841 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:19.574182 17846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:19.574402 17846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 LEADER]: Becoming Leader. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Running, Role: LEADER
I20260620 07:14:19.574753 17846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:19.575366 17841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:19.576529 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:19.576685 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:19.577005 17848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc7ff63f69d4b518aa8c240edd3064f. Latest consensus state: current_term: 1 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:19.577116 17848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:19.577147 17855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:19.580372 17855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:19.587141 17855 catalog_manager.cc:1347] Generated new cluster ID: 77f460744d104a2b836bbbd31949cdb1
I20260620 07:14:19.587257 17855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:19.600369 17855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:19.601759 17855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:19.614252 17855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Generated new TSK 0
I20260620 07:14:19.615155 17855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:14:19.655804 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:19.770789 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:19.771128 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:19.771209 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:19.775553 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:19.775647 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:19.775754 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:19.781785 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:19.782974 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:19.784073 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:19.794628 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.794549 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:19.794580 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:19.794561 17867 server_base.cc:1034] running on GCE node
I20260620 07:14:19.795394 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:19.796165 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:19.797399 17867 hybrid_clock.cc:648] HybridClock initialized: now 1781939659797385 us; error 28 us; skew 500 ppm
Jun 20 07:14:19 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:19.800484 17867 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:19.801581 17867 webserver.cc:460] Webserver started at http://127.3.245.1:39141/ using document root <none> and password file <none>
I20260620 07:14:19.802191 17867 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:19.802294 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:19.802536 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:19.804876 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "a2cefb2543164df79070ee05a7aeb48f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "921782025376a791dfc260b8f90163da"
server_key_iv: "cb007b37c66c7bcb535d3e88db870619"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.805503 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "a2cefb2543164df79070ee05a7aeb48f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "921782025376a791dfc260b8f90163da"
server_key_iv: "cb007b37c66c7bcb535d3e88db870619"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.809548 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:14:19.812155 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.813501 17867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:14:19.813645 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "a2cefb2543164df79070ee05a7aeb48f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "921782025376a791dfc260b8f90163da"
server_key_iv: "cb007b37c66c7bcb535d3e88db870619"
server_key_version: "encryptionkey@0"
I20260620 07:14:19.813773 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:19.829051 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:19.832784 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:19.833005 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:19.833639 17867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:19.834509 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:19.834580 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.834653 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:19.834703 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:19.844324 17867 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37425
I20260620 07:14:19.844350 17995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37425 every 8 connection(s)
I20260620 07:14:19.845376 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:19.853241 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 17867
I20260620 07:14:19.853423 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:19.853719 4052 external_mini_cluster.cc:1428] Setting key b83da828795c8dbbf5e84a92d32b49f0
Jun 20 07:14:19 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939659, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:19.859880 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:19.847411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:54999 (local address 127.3.245.62:45645)
0620 07:14:19.847604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:19.847611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:19.848943 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 07:14:19.849868 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:19.849872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:19.849941 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:19.850018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:19.851239 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.851792 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.852966 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.853234 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.853356 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:19.856117 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:19.856121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:19.856124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:19.856159 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:19.857727 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.858305 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.858310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.858312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.858395 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:19.858750 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:19.858763 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:19.858765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:19.858952 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:19.859060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:19.859572 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 07:14:19.859695 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:14:19.860754 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:19.847888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:54999)
0620 07:14:19.848789 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:19.848833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:19.849654 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:19.850191 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:19.850198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:19.850587 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:19.851078 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:19.851091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.851924 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.851927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:19.852835 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:19.852845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:19.853413 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:19.853417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:19.853621 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:19.854296 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:19.854312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:19.855944 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:19.857898 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:19.857905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:19.857922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:19.858192 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:19.858499 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:19.858502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:19.858504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:19.858655 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:19.859059 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:19.859068 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:19.859329 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 07:14:19.859519 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 07:14:19.859855 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":158,"threads_started":1}
I20260620 07:14:19.862021 17996 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:19.862272 17996 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:19.862788 17996 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:14:19.864459 17804 ts_manager.cc:194] Registered new tserver with Master: a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425)
I20260620 07:14:19.865718 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:54999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:14:19.883669 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:19.998014 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:19.998342 18001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:19.998433 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:20.002759 18001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:20.002835 18001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:20.002943 18001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:20.008563 18001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:20.009794 18001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:20.011168 18001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:20.021500 18007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.021533 18009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.021502 18006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:20.022032 18001 server_base.cc:1034] running on GCE node
I20260620 07:14:20.022464 18001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:20.023316 18001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:20.024494 18001 hybrid_clock.cc:648] HybridClock initialized: now 1781939660024463 us; error 43 us; skew 500 ppm
Jun 20 07:14:20 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:20.027866 18001 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:20.029210 18001 webserver.cc:460] Webserver started at http://127.3.245.2:33163/ using document root <none> and password file <none>
I20260620 07:14:20.029973 18001 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:20.030067 18001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:20.030339 18001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:20.033181 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "3bc93a3a0cdef469863e9d4026e20eed"
server_key_iv: "e3ae7f3c74751f569cc4c5d1a60dab32"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.033808 18001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "3bc93a3a0cdef469863e9d4026e20eed"
server_key_iv: "e3ae7f3c74751f569cc4c5d1a60dab32"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.037974 18001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:14:20.040954 18016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.042266 18001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:20.042543 18001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "3bc93a3a0cdef469863e9d4026e20eed"
server_key_iv: "e3ae7f3c74751f569cc4c5d1a60dab32"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.042680 18001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:20.067155 18001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:20.070816 18001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:20.071013 18001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:20.071614 18001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:20.072467 18001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:20.072540 18001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.072634 18001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:20.072688 18001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.082423 18001 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:39961
I20260620 07:14:20.082465 18129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:39961 every 8 connection(s)
I20260620 07:14:20.083384 18001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:20.091122 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18001
I20260620 07:14:20.091257 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:20.091638 4052 external_mini_cluster.cc:1428] Setting key 11e3101026f4de43ac14b76a0cc824c7
Jun 20 07:14:20 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:20.099186 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.085495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:40179 (local address 127.3.245.62:45645)
0620 07:14:20.085639 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.085642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.086721 (+ 1079us) server_negotiation.cc:407] Connection header received
0620 07:14:20.087608 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.087611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.087664 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.087760 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.089185 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.089782 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.090979 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.091362 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.091503 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.094598 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.094605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.094612 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.094646 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.096346 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.097137 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.097140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.097142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.097237 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.097712 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.097715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.097716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.098036 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.098128 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.098927 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.099056 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:14:20.100569 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.085886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:40179)
0620 07:14:20.086576 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.086618 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.087396 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.088001 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.088009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.088397 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:20.089006 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.089021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.089924 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.089927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.090832 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.090842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.091556 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.091559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.091724 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.092604 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:20.092662 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:20.094416 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:20.096474 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.096485 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.096507 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.097000 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.097374 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.097381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.097386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.097600 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.098169 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:20.098180 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:20.098586 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.098825 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.099260 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":111,"threads_started":1}
I20260620 07:14:20.102021 18130 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:20.102322 18130 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:14:20.102918 18130 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
I20260620 07:14:20.104267 17804 ts_manager.cc:194] Registered new tserver with Master: 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961)
I20260620 07:14:20.105014 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:40179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:14:20.124136 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:20.241271 18135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:20.241550 18135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:20.241595 18135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:20.245800 18135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:20.245880 18135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:20.245944 18135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:20.251559 18135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:20.252712 18135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:20.253727 18135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:20.263882 18140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:20.263981 18135 server_base.cc:1034] running on GCE node
W20260620 07:14:20.263862 18141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.263870 18143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:20.264551 18135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:20.265292 18135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:20.266480 18135 hybrid_clock.cc:648] HybridClock initialized: now 1781939660266463 us; error 31 us; skew 500 ppm
Jun 20 07:14:20 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:20.269474 18135 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:20.270527 18135 webserver.cc:460] Webserver started at http://127.3.245.3:32843/ using document root <none> and password file <none>
I20260620 07:14:20.271096 18135 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:20.271186 18135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:20.271428 18135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:20.273599 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "d3d85528e46f8926d981ccd426dcffc2"
server_key_iv: "79f777039ef1b6d4952ef5f4bb868548"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.274156 18135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "d3d85528e46f8926d981ccd426dcffc2"
server_key_iv: "79f777039ef1b6d4952ef5f4bb868548"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.278458 18135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:20.280941 18150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.282300 18135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 07:14:20.282402 18135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "d3d85528e46f8926d981ccd426dcffc2"
server_key_iv: "79f777039ef1b6d4952ef5f4bb868548"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.282492 18135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:20.296373 18135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:20.299853 18135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:20.300057 18135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:20.300604 18135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:20.301534 18135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:20.301582 18135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.301622 18135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:20.301679 18135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.310930 18135 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:40533
I20260620 07:14:20.310950 18263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:40533 every 8 connection(s)
I20260620 07:14:20.311949 18135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:20.321655 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18135
I20260620 07:14:20.321810 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:20.322168 4052 external_mini_cluster.cc:1428] Setting key f9f27f02ce45a30cf3abe6fe0cf6d5e8
Jun 20 07:14:20 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:20.327545 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.313931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:58641 (local address 127.3.245.62:45645)
0620 07:14:20.314134 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.314140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.315277 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 07:14:20.316195 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.316198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.316250 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.316338 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.317904 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.318441 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.319810 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.320087 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.320182 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.323294 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.323297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.323300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.323334 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.325345 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.325984 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.325987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.325988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.326074 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.326425 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.326428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.326429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.326625 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.326708 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.327295 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.327412 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 07:14:20.328411 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.314364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:58641)
0620 07:14:20.315116 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.315159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.315977 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.316539 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.316552 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.317046 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:20.317721 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.317739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.318588 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.318592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.319649 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.319660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.320242 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.320245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.320407 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.321097 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:20.321115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:20.323084 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:20.325494 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.325500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.325514 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.325878 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.326167 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:20.326170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:20.326172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:20.326330 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:20.326750 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:20.326756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:20.327058 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.327231 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.327514 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":123,"threads_started":1}
I20260620 07:14:20.329749 18264 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:20.330042 18264 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:20.330610 18264 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
I20260620 07:14:20.331681 17804 ts_manager.cc:194] Registered new tserver with Master: b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
I20260620 07:14:20.332278 17804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:58641
I20260620 07:14:20.336337 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:20.346410 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.338476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.3.245.62:45645)
0620 07:14:20.338616 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.338620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.338762 (+ 142us) server_negotiation.cc:407] Connection header received
0620 07:14:20.338854 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.338856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.338897 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.338962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:20.339629 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.340160 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.341184 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.341430 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.341532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.342898 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:20.342901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:20.342904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:20.342928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:20.344574 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.345097 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.345100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.345101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.345180 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:20.345456 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:20.345459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:20.345460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:20.345697 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:20.345872 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.346088 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.346190 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:14:20.349906 17804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:20.353199 17804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:20.370748 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.362762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.3.245.3:40533)
0620 07:14:20.363187 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.363200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.363277 (+ 77us) server_negotiation.cc:407] Connection header received
0620 07:14:20.363586 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.363594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.363826 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.364016 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.364549 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.365958 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.367760 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.369083 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.369230 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.369432 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.370411 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.370537 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":118,"threads_started":1}
I20260620 07:14:20.371212 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.362576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40533 (local address 127.0.0.1:47130)
0620 07:14:20.363108 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.363158 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.363334 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.364244 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.364251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.364276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.364437 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.364448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.366130 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.366136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.367620 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.367631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.369507 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.369511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.369658 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.369744 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.369870 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.371055 (+ 1185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":106,"threads_started":1}
I20260620 07:14:20.372700 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.362793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39961 (local address 127.0.0.1:37294)
0620 07:14:20.364166 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.364181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.364348 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.365088 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.365090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.365109 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.365214 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.365220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.369170 (+ 3950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.369175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.370494 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.370506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.372239 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.372242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.372336 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.372358 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.372460 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.372527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"spinlock_wait_cycles":896,"thread_start_us":110,"threads_started":1}
I20260620 07:14:20.372824 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.363470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.3.245.2:39961)
0620 07:14:20.363844 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.363856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.364265 (+ 409us) server_negotiation.cc:407] Connection header received
0620 07:14:20.364518 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.364525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.364729 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.364853 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.365341 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.368981 (+ 3640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.370658 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.371856 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.371967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.372142 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.372509 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.372666 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":83,"threads_started":1}
I20260620 07:14:20.375391 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.363346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37425 (local address 127.0.0.1:58914)
0620 07:14:20.364285 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.364299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.364391 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.365854 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.365857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.365876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.365986 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.365995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.372173 (+ 6178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.372179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.373519 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.373530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.374955 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.374959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.375056 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.375079 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.375183 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.375259 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":108,"threads_started":1}
I20260620 07:14:20.375546 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.363764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.3.245.1:37425)
0620 07:14:20.365112 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.365119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.365169 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:14:20.365285 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.365293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.365568 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.365714 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.367809 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.369122 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.373664 (+ 4542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.374605 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.374715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.374867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.375221 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.375374 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
I20260620 07:14:20.374859 18198 tablet_service.cc:1467] Processing CreateTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 (DEFAULT_TABLE table=test-table [id=5f84628cc16f4a578e6458d72d9e23ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.376075 18064 tablet_service.cc:1467] Processing CreateTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 (DEFAULT_TABLE table=test-table [id=5f84628cc16f4a578e6458d72d9e23ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.376881 18198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4619dbb594014d85b3b17fc53ff2ee44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.377266 18064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4619dbb594014d85b3b17fc53ff2ee44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.378810 17930 tablet_service.cc:1467] Processing CreateTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 (DEFAULT_TABLE table=test-table [id=5f84628cc16f4a578e6458d72d9e23ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:20.379904 17930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4619dbb594014d85b3b17fc53ff2ee44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:20.383158 18281 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Bootstrap starting.
I20260620 07:14:20.385104 18282 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Bootstrap starting.
I20260620 07:14:20.386126 18281 tablet_bootstrap.cc:654] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.387072 18281 log.cc:824] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.388048 18282 tablet_bootstrap.cc:654] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.388887 18282 log.cc:824] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.389392 18281 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: No bootstrap required, opened a new log
I20260620 07:14:20.389619 18281 ts_tablet_manager.cc:1397] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260620 07:14:20.390926 18282 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: No bootstrap required, opened a new log
I20260620 07:14:20.391108 18282 ts_tablet_manager.cc:1397] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:14:20.392926 18285 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Bootstrap starting.
I20260620 07:14:20.393639 18281 raft_consensus.cc:348] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.393934 18281 raft_consensus.cc:374] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.393993 18281 raft_consensus.cc:729] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b445fa8bcaa14c5c8c91207fe930ea56, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.394510 18281 consensus_queue.cc:260] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.394959 18282 raft_consensus.cc:348] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.395216 18282 raft_consensus.cc:374] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.395215 18264 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
I20260620 07:14:20.395269 18282 raft_consensus.cc:729] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3192261f5cc84ae28ebfec8ccbb61e0e, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.395579 18281 ts_tablet_manager.cc:1428] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 07:14:20.395676 18282 consensus_queue.cc:260] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.395815 18285 tablet_bootstrap.cc:654] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:20.396373 18130 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
I20260620 07:14:20.396643 18282 ts_tablet_manager.cc:1428] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 07:14:20.397531 18285 log.cc:824] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:20.399606 18285 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: No bootstrap required, opened a new log
I20260620 07:14:20.399848 18285 ts_tablet_manager.cc:1397] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 07:14:20.403693 18285 raft_consensus.cc:348] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.403930 18285 raft_consensus.cc:374] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:20.403999 18285 raft_consensus.cc:729] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cefb2543164df79070ee05a7aeb48f, State: Initialized, Role: FOLLOWER
I20260620 07:14:20.404459 18285 consensus_queue.cc:260] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.405172 17996 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
I20260620 07:14:20.405388 18285 ts_tablet_manager.cc:1428] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 07:14:20.562978 18265 tablet.cc:2369] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:20.584563 18131 tablet.cc:2369] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:14:20.596889 17997 tablet.cc:2369] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:20.689579 18289 raft_consensus.cc:482] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:20.689890 18289 raft_consensus.cc:504] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.691210 18289 leader_election.cc:288] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533), 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961)
I20260620 07:14:20.695539 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.691612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:58585 (local address 127.3.245.3:40533)
0620 07:14:20.691769 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.691775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.691990 (+ 215us) server_negotiation.cc:407] Connection header received
0620 07:14:20.692102 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.692105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.692165 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.692258 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.692604 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.693168 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.694497 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.694916 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.695035 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.695106 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.695335 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.695406 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:14:20.695540 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.691656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:45683 (local address 127.3.245.2:39961)
0620 07:14:20.691848 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.691854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.692115 (+ 261us) server_negotiation.cc:407] Connection header received
0620 07:14:20.692245 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.692252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.692304 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.692389 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:20.692739 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.693305 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.694528 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.694914 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.695028 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.695107 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.695330 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.695410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 07:14:20.695573 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.691547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39961 (local address 127.3.245.1:45683)
0620 07:14:20.691996 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.692011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.692121 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.692421 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.692424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.692437 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.692598 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.692604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.693432 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.693435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.694286 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.694295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.695043 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.695047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.695191 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.695218 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.695316 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.695396 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":59,"thread_start_us":121,"threads_started":1}
I20260620 07:14:20.695583 18290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.691507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40533 (local address 127.3.245.1:58585)
0620 07:14:20.691866 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.691883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:20.692009 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:20.692272 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:20.692275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:20.692291 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:20.692476 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.692484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.693280 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.693283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.694266 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:20.694275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.695043 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.695047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:20.695194 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.695221 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:14:20.695314 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:14:20.695411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":78,"thread_start_us":107,"threads_started":1}
I20260620 07:14:20.696460 18084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4619dbb594014d85b3b17fc53ff2ee44" candidate_uuid: "a2cefb2543164df79070ee05a7aeb48f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" is_pre_election: true
I20260620 07:14:20.696581 18218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4619dbb594014d85b3b17fc53ff2ee44" candidate_uuid: "a2cefb2543164df79070ee05a7aeb48f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" is_pre_election: true
I20260620 07:14:20.696847 18084 raft_consensus.cc:2454] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cefb2543164df79070ee05a7aeb48f in term 0.
I20260620 07:14:20.696931 18218 raft_consensus.cc:2454] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2cefb2543164df79070ee05a7aeb48f in term 0.
I20260620 07:14:20.697417 17884 leader_election.cc:302] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3192261f5cc84ae28ebfec8ccbb61e0e, a2cefb2543164df79070ee05a7aeb48f, b445fa8bcaa14c5c8c91207fe930ea56; no voters:
I20260620 07:14:20.697731 18289 raft_consensus.cc:2781] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:20.697829 18289 raft_consensus.cc:482] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:20.697861 18289 raft_consensus.cc:3037] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.698937 18289 raft_consensus.cc:504] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.699316 18289 leader_election.cc:288] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 1 election: Requested vote from peers b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533), 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961)
I20260620 07:14:20.699774 18218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4619dbb594014d85b3b17fc53ff2ee44" candidate_uuid: "a2cefb2543164df79070ee05a7aeb48f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
I20260620 07:14:20.699913 18218 raft_consensus.cc:3037] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.701107 18218 raft_consensus.cc:2454] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cefb2543164df79070ee05a7aeb48f in term 1.
I20260620 07:14:20.701483 18084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4619dbb594014d85b3b17fc53ff2ee44" candidate_uuid: "a2cefb2543164df79070ee05a7aeb48f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
I20260620 07:14:20.701531 17883 leader_election.cc:302] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f, b445fa8bcaa14c5c8c91207fe930ea56; no voters:
I20260620 07:14:20.701633 18084 raft_consensus.cc:3037] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:20.701812 18289 raft_consensus.cc:2781] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:20.702095 18289 raft_consensus.cc:686] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 LEADER]: Becoming Leader. State: Replica: a2cefb2543164df79070ee05a7aeb48f, State: Running, Role: LEADER
I20260620 07:14:20.702536 18289 consensus_queue.cc:237] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:20.702926 18084 raft_consensus.cc:2454] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2cefb2543164df79070ee05a7aeb48f in term 1.
I20260620 07:14:20.706004 17804 catalog_manager.cc:5495] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f reported cstate change: term changed from 0 to 1, leader changed from <none> to a2cefb2543164df79070ee05a7aeb48f (127.3.245.1). New cstate: current_term: 1 leader_uuid: "a2cefb2543164df79070ee05a7aeb48f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } health_report { overall_health: HEALTHY } } }
I20260620 07:14:20.735498 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.732067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.3.245.1:37425)
0620 07:14:20.732207 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:20.732212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:20.732226 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:14:20.732304 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:20.732306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:20.732362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:20.732450 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:20.732771 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.733298 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.734160 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:20.734378 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:20.734461 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:20.734747 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:20.735171 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:20.735272 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:20.735335 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 07:14:20.735387 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:14:20.741760 18084 raft_consensus.cc:1261] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Refusing update from remote peer a2cefb2543164df79070ee05a7aeb48f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:20.741760 18218 raft_consensus.cc:1261] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Refusing update from remote peer a2cefb2543164df79070ee05a7aeb48f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:20.742543 18292 consensus_queue.cc:1035] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:20.742746 18289 consensus_queue.cc:1035] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:20.760558 17805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:20.760828 17805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:20.763504 17805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Sending DeleteTablet for 3 replicas of tablet 4619dbb594014d85b3b17fc53ff2ee44
I20260620 07:14:20.764386 18198 tablet_service.cc:1514] Processing DeleteTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:47130
I20260620 07:14:20.764533 18064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:37294
I20260620 07:14:20.764544 17930 tablet_service.cc:1514] Processing DeleteTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:58914
I20260620 07:14:20.765136 18307 tablet_replica.cc:330] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: stopping tablet replica
I20260620 07:14:20.765250 18306 tablet_replica.cc:330] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: stopping tablet replica
I20260620 07:14:20.765416 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17867
I20260620 07:14:20.765501 18307 raft_consensus.cc:2229] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:20.765578 18306 raft_consensus.cc:2229] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:20.765879 18306 raft_consensus.cc:2258] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:20.766310 18305 tablet_replica.cc:330] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: stopping tablet replica
I20260620 07:14:20.766592 18305 raft_consensus.cc:2229] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:20.766916 18305 raft_consensus.cc:2258] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:20.767737 18306 ts_tablet_manager.cc:1905] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:20.768488 18305 ts_tablet_manager.cc:1905] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:20.770059 18306 ts_tablet_manager.cc:1918] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:20.770133 18306 log.cc:1196] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/4619dbb594014d85b3b17fc53ff2ee44
I20260620 07:14:20.770387 18306 ts_tablet_manager.cc:1939] T 4619dbb594014d85b3b17fc53ff2ee44 P b445fa8bcaa14c5c8c91207fe930ea56: Deleting consensus metadata
I20260620 07:14:20.771533 17789 catalog_manager.cc:4841] TS b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): tablet 4619dbb594014d85b3b17fc53ff2ee44 (table test-table [id=5f84628cc16f4a578e6458d72d9e23ea]) successfully deleted
I20260620 07:14:20.771905 18305 ts_tablet_manager.cc:1918] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:20.772017 18305 log.cc:1196] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/4619dbb594014d85b3b17fc53ff2ee44
I20260620 07:14:20.772368 18305 ts_tablet_manager.cc:1939] T 4619dbb594014d85b3b17fc53ff2ee44 P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting consensus metadata
W20260620 07:14:20.773532 17790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37425 (error 108)
I20260620 07:14:20.773895 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18001
I20260620 07:14:20.774129 17790 catalog_manager.cc:4841] TS 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): tablet 4619dbb594014d85b3b17fc53ff2ee44 (table test-table [id=5f84628cc16f4a578e6458d72d9e23ea]) successfully deleted
I20260620 07:14:20.774830 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:20.774453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37425 (local address 127.0.0.1:58934)
0620 07:14:20.774683 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:20.774750 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 07:14:20.775066 17790 catalog_manager.cc:4584] TS a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4619dbb594014d85b3b17fc53ff2ee44: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37425: connect: Connection refused (error 111)
I20260620 07:14:20.780684 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18135
I20260620 07:14:20.787036 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 17773
I20260620 07:14:20.793740 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--webserver_interface=127.3.245.62
--webserver_port=36723
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:20.909948 18309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:20.910290 18309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:20.910389 18309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:20.914789 18309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:20.914892 18309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:20.914938 18309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:20.914997 18309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:20.915053 18309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:20.920961 18309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=36723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:20.922102 18309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:20.923170 18309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:20.936283 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.936264 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:20.936264 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:20.936756 18309 server_base.cc:1034] running on GCE node
I20260620 07:14:20.937465 18309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:20.938663 18309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:20.939822 18309 hybrid_clock.cc:648] HybridClock initialized: now 1781939660939808 us; error 39 us; skew 500 ppm
Jun 20 07:14:20 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:20.942883 18309 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:20.943917 18309 webserver.cc:460] Webserver started at http://127.3.245.62:36723/ using document root <none> and password file <none>
I20260620 07:14:20.944535 18309 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:20.944669 18309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:20.948967 18309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 07:14:20.951411 18325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:20.952922 18309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:20.953068 18309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "0bc7ff63f69d4b518aa8c240edd3064f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "6f059a5658f5dcc23918fb59d4d668bb"
server_key_iv: "79a87024134cd50e995304b74059223d"
server_key_version: "encryptionkey@0"
I20260620 07:14:20.953697 18309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:20.968071 18309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:20.971675 18309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:20.971905 18309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:20.980752 18309 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45645
I20260620 07:14:20.981088 18377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45645 every 8 connection(s)
I20260620 07:14:20.982038 18309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:20.986543 18378 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:14:20.988226 18378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap starting.
I20260620 07:14:20.990904 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18309
I20260620 07:14:20.991365 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:37425
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=39141
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:21.005980 18378 log.cc:824] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:21.014940 18378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:14:21.015425 18378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap complete.
I20260620 07:14:21.019560 18378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:21.019971 18378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Initialized, Role: FOLLOWER
I20260620 07:14:21.020473 18378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:21.020589 18378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:21.020713 18378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:21.020859 18378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:14:21.022866 18378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:21.023216 18378 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc7ff63f69d4b518aa8c240edd3064f; no voters:
I20260620 07:14:21.023540 18378 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:14:21.023694 18382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:14:21.024021 18382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 LEADER]: Becoming Leader. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Running, Role: LEADER
I20260620 07:14:21.024349 18382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:21.024940 18378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:21.025804 18384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc7ff63f69d4b518aa8c240edd3064f. Latest consensus state: current_term: 2 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:21.025954 18384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:21.026228 18383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:21.026317 18383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:21.027038 18391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:21.032064 18391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:21.032727 18391 catalog_manager.cc:1259] Loaded cluster ID: 77f460744d104a2b836bbbd31949cdb1
I20260620 07:14:21.032789 18391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:21.034228 18391 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 07:14:21.038897 18391 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 07:14:21.044732 18391 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:21.045606 18391 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Loaded TSK: 0
I20260620 07:14:21.046872 18391 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:14:21.116570 18380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:21.116957 18380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:21.117043 18380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:21.121404 18380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:21.121480 18380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:21.121541 18380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:21.127424 18380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=39141
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:21.128571 18380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:21.129736 18380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:21.140100 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.140281 18380 server_base.cc:1034] running on GCE node
W20260620 07:14:21.140401 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.140107 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.141002 18380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:21.141899 18380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:21.143117 18380 hybrid_clock.cc:648] HybridClock initialized: now 1781939661143101 us; error 28 us; skew 500 ppm
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:21.147362 18380 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:21.148726 18380 webserver.cc:460] Webserver started at http://127.3.245.1:39141/ using document root <none> and password file <none>
I20260620 07:14:21.149515 18380 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:21.149636 18380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:21.155129 18380 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:21.158166 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.159816 18380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:14:21.159966 18380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "a2cefb2543164df79070ee05a7aeb48f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "921782025376a791dfc260b8f90163da"
server_key_iv: "cb007b37c66c7bcb535d3e88db870619"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.160737 18380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:21.177002 18380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:21.181592 18380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:21.181814 18380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:21.182510 18380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:21.183868 18422 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:14:21.188329 18380 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:14:21.188414 18380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 07:14:21.188480 18380 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:14:21.190614 18380 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:14:21.190683 18380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:14:21.191073 18422 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Bootstrap starting.
I20260620 07:14:21.202181 18380 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37425
I20260620 07:14:21.202255 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37425 every 8 connection(s)
I20260620 07:14:21.203289 18380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:21.208966 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18380
I20260620 07:14:21.209468 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:39961
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=33163
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:21.210688 18422 log.cc:824] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:21.221233 18422 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:14:21.221349 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.205603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:37925 (local address 127.3.245.62:45645)
0620 07:14:21.206044 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.206073 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.206436 (+ 363us) server_negotiation.cc:407] Connection header received
0620 07:14:21.207590 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.207600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.207913 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.208255 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.209771 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.210770 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.212656 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.213160 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.213368 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.216012 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.216017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.216030 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.216064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.217937 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.218535 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.218540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.218544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.218674 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.219030 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.219035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.219037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.219409 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.219559 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.220120 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.220391 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260620 07:14:21.221752 18422 tablet_bootstrap.cc:492] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Bootstrap complete.
I20260620 07:14:21.221928 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.205708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:37925)
0620 07:14:21.206267 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.206312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.207145 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.208328 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.208336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.208850 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:21.209411 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.209428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.210960 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.210965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.212486 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.212499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.213331 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.213335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.213535 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.214179 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:21.214196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:21.215776 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:21.218081 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.218087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.218101 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.218388 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.218779 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.218784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.218786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.218930 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.219532 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:21.219538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:21.219803 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.219991 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.220369 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":160,"threads_started":1}
I20260620 07:14:21.223320 18422 ts_tablet_manager.cc:1397] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Time spent bootstrapping tablet: real 0.033s user 0.030s sys 0.000s
I20260620 07:14:21.224460 18530 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:21.224781 18530 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:21.225349 18530 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
I20260620 07:14:21.226518 18422 raft_consensus.cc:348] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.227057 18422 raft_consensus.cc:729] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cefb2543164df79070ee05a7aeb48f, State: Initialized, Role: FOLLOWER
I20260620 07:14:21.227574 18422 consensus_queue.cc:260] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.228092 18342 ts_manager.cc:194] Registered new tserver with Master: a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425)
I20260620 07:14:21.228694 18422 ts_tablet_manager.cc:1428] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 07:14:21.231804 18342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:37925
I20260620 07:14:21.234511 18530 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:14:21.243278 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.231334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.3.245.1:37425)
0620 07:14:21.232670 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.232682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.232706 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:14:21.232774 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.232780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.232985 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.233118 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.233895 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.235040 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.236223 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.236714 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.236836 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.239296 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.239306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.239317 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.239343 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.241082 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.241695 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.241700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.241702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.241791 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.242131 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.242134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.242135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.242425 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.242541 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.242986 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.243140 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":121,"threads_started":1}
I20260620 07:14:21.243284 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.231238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37425 (local address 127.0.0.1:58944)
0620 07:14:21.232105 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.232153 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.232358 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.233151 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.233157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.233610 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:21.233754 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.233763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.235159 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.235162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.235985 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.235995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.236837 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.236840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.236979 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.237584 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:21.237596 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:21.239084 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:21.241213 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.241224 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.241230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.241548 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.241905 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.241908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.241910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.242010 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.242562 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:21.242571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:21.242714 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.242867 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.243091 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":125,"threads_started":1}
I20260620 07:14:21.244525 18464 tablet_service.cc:1514] Processing DeleteTablet for tablet 4619dbb594014d85b3b17fc53ff2ee44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:20 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:58944
I20260620 07:14:21.245596 18539 tablet_replica.cc:330] stopping tablet replica
I20260620 07:14:21.245899 18539 raft_consensus.cc:2229] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:21.246152 18539 raft_consensus.cc:2258] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:21.247167 18539 ts_tablet_manager.cc:1905] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:21.250846 18539 ts_tablet_manager.cc:1918] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:21.250978 18539 log.cc:1196] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/4619dbb594014d85b3b17fc53ff2ee44
I20260620 07:14:21.251262 18539 ts_tablet_manager.cc:1939] T 4619dbb594014d85b3b17fc53ff2ee44 P a2cefb2543164df79070ee05a7aeb48f: Deleting consensus metadata
I20260620 07:14:21.253558 18327 catalog_manager.cc:4841] TS a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425): tablet 4619dbb594014d85b3b17fc53ff2ee44 (table test-table [id=5f84628cc16f4a578e6458d72d9e23ea]) successfully deleted
W20260620 07:14:21.337137 18534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:21.337458 18534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:21.337539 18534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:21.341878 18534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:21.341955 18534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:21.342054 18534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:21.347757 18534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:39961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=33163
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:21.348963 18534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:21.350039 18534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:21.360240 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.360338 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.360265 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.360447 18534 server_base.cc:1034] running on GCE node
I20260620 07:14:21.361003 18534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:21.361716 18534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:21.362900 18534 hybrid_clock.cc:648] HybridClock initialized: now 1781939661362884 us; error 28 us; skew 500 ppm
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:21.366089 18534 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:21.367148 18534 webserver.cc:460] Webserver started at http://127.3.245.2:33163/ using document root <none> and password file <none>
I20260620 07:14:21.367765 18534 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:21.367844 18534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:21.372171 18534 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:21.374563 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.376070 18534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:21.376242 18534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "3bc93a3a0cdef469863e9d4026e20eed"
server_key_iv: "e3ae7f3c74751f569cc4c5d1a60dab32"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.376989 18534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:21.399706 18534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:21.403458 18534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:21.403710 18534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:21.404379 18534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:21.405325 18534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:21.405401 18534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.405475 18534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:21.405527 18534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.415506 18534 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:39961
I20260620 07:14:21.415526 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:39961 every 8 connection(s)
I20260620 07:14:21.416786 18534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:21.418275 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18534
I20260620 07:14:21.418721 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:40533
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=32843
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:21.433830 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.418991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:49297 (local address 127.3.245.62:45645)
0620 07:14:21.419173 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.419180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.420351 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 07:14:21.421338 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.421344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.421413 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.421529 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.422851 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.423810 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.425214 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.425672 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.425818 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.429158 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.429165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.429176 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.429222 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.431128 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.431770 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.431774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.431776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.431879 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.432347 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.432350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.432352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.432584 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.432705 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.433482 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.433633 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:14:21.434862 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.419391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:49297)
0620 07:14:21.420165 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.420221 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.421112 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.421716 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.421723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.422165 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:21.422673 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.422687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.423978 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.423981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.425087 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.425098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.425847 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.425850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.426195 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.426848 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:21.426866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:21.428909 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:21.431307 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.431314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.431330 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.431645 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.432016 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.432019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.432021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.432206 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.432769 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:21.432777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:21.433149 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.433367 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.433743 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":156,"threads_started":1}
I20260620 07:14:21.436601 18668 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:21.436954 18668 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:21.437659 18668 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
I20260620 07:14:21.439079 18342 ts_manager.cc:194] Registered new tserver with Master: 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961)
I20260620 07:14:21.439694 18342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:49297
W20260620 07:14:21.540073 18671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:21.540432 18671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:21.540529 18671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:21.544950 18671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:21.545053 18671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:21.545153 18671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:21.551118 18671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:40533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=32843
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:21.552340 18671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:21.553505 18671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:21.564536 18677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.564545 18676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.564594 18679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.564859 18671 server_base.cc:1034] running on GCE node
I20260620 07:14:21.565519 18671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:21.566313 18671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:21.567538 18671 hybrid_clock.cc:648] HybridClock initialized: now 1781939661567517 us; error 31 us; skew 500 ppm
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:21.570799 18671 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:21.571810 18671 webserver.cc:460] Webserver started at http://127.3.245.3:32843/ using document root <none> and password file <none>
I20260620 07:14:21.572422 18671 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:21.572541 18671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:21.576727 18671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:21.579106 18686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.580817 18671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:21.580931 18671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "d3d85528e46f8926d981ccd426dcffc2"
server_key_iv: "79f777039ef1b6d4952ef5f4bb868548"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.581492 18671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:21.594596 18671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:21.598093 18671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:21.598312 18671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:21.598906 18671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:21.599757 18671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:21.599808 18671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.599852 18671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:21.599869 18671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.610160 18671 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:40533
I20260620 07:14:21.610183 18799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:40533 every 8 connection(s)
I20260620 07:14:21.611379 18671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:21.616775 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18671
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:21.626367 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.617543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.3.245.62:45645)
0620 07:14:21.617799 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.617802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.618071 (+ 269us) server_negotiation.cc:407] Connection header received
0620 07:14:21.618190 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.618192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.618234 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.618311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.619117 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.619611 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.620480 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.620825 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.620941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.622415 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.622420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.622423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.622452 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.624449 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.624962 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.624965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.624966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.625042 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.625368 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.625372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.625373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.625682 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.625921 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.626069 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.626200 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":110,"threads_started":1}
I20260620 07:14:21.627697 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.613539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47989 (local address 127.3.245.62:45645)
0620 07:14:21.613779 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.613783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.615017 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 07:14:21.615890 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.615893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.615944 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.616029 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:21.617740 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.618519 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.619815 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.620050 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.620170 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.623376 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:21.623379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:21.623381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:21.623426 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:21.625042 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.625914 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.625917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.625919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.626010 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:21.626442 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:21.626448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:21.626452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:21.626670 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:21.626751 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.627460 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.627589 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 07:14:21.628680 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.613987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:47989)
0620 07:14:21.614831 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.614877 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.615666 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.616318 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.616325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.616823 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:21.617543 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.617569 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.618720 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.618723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.619641 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.619650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.620208 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.620212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.620390 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.621070 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:21.621092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:21.623193 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:21.625384 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.625394 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.625418 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.625802 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.626116 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:21.626119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:21.626121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:21.626319 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:21.626833 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:21.626839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:21.627176 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.627372 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.627647 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":147,"threads_started":1}
I20260620 07:14:21.629962 18800 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45645
I20260620 07:14:21.630185 18800 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:21.630661 18800 heartbeater.cc:511] Master 127.3.245.62:45645 requested a full tablet report, sending...
I20260620 07:14:21.631651 18342 ts_manager.cc:194] Registered new tserver with Master: b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
I20260620 07:14:21.632179 18342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:47989
I20260620 07:14:21.640839 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:21.645545 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.641492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.3.245.62:45645)
0620 07:14:21.641684 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.641689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.641843 (+ 154us) server_negotiation.cc:407] Connection header received
0620 07:14:21.642050 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.642052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.642174 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.642327 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:21.642576 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.643098 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.644003 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.644232 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.644345 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.644717 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:21.645133 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:21.645274 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.645336 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.645390 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 07:14:21.646250 18342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:21.649600 18342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:21.658121 18464 tablet_service.cc:1467] Processing CreateTablet for tablet 33e4a11e5f444b968f70470cf398354b (DEFAULT_TABLE table=test-table [id=5b296ca7c0194ee9bf61fd54cad9aa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:21.658805 18464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e4a11e5f444b968f70470cf398354b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:21.662127 18422 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Bootstrap starting.
I20260620 07:14:21.662271 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.657126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40533 (local address 127.0.0.1:47146)
0620 07:14:21.657639 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.657652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.657749 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.658343 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.658346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.658363 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:21.658514 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.658523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.660025 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.660028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.661022 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.661032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.661856 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.661860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.661942 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.661985 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.662054 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.662126 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":132,"threads_started":1}
I20260620 07:14:21.662305 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.657425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.3.245.3:40533)
0620 07:14:21.657918 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.657927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.657946 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:14:21.658010 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.658014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.658194 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.658414 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:21.658824 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.659853 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.661155 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.661724 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.661862 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.662031 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.662086 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.662177 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":154,"threads_started":1}
I20260620 07:14:21.663108 18422 tablet_bootstrap.cc:654] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:21.663353 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.656920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39961 (local address 127.0.0.1:37302)
0620 07:14:21.657115 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.657133 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.657226 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.658603 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.658606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.658627 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:21.658714 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.658720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.660249 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.660254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.661426 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.661437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.662979 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.662983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.663079 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.663103 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.663164 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.663234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 07:14:21.663780 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.657450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.3.245.2:39961)
0620 07:14:21.657935 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.657946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.657969 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:14:21.658042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.658048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.658255 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.658634 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:21.658795 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.660040 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.661588 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.662851 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.662975 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.663229 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.663298 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.663630 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":92,"threads_started":1}
I20260620 07:14:21.664865 18422 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: No bootstrap required, opened a new log
I20260620 07:14:21.664965 18422 ts_tablet_manager.cc:1397] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:21.665443 18422 raft_consensus.cc:348] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.665570 18422 raft_consensus.cc:374] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:21.665609 18422 raft_consensus.cc:729] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cefb2543164df79070ee05a7aeb48f, State: Initialized, Role: FOLLOWER
I20260620 07:14:21.665778 18422 consensus_queue.cc:260] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.666050 18422 ts_tablet_manager.cc:1428] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:14:21.666185 18734 tablet_service.cc:1467] Processing CreateTablet for tablet 33e4a11e5f444b968f70470cf398354b (DEFAULT_TABLE table=test-table [id=5b296ca7c0194ee9bf61fd54cad9aa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:21.666878 18602 tablet_service.cc:1467] Processing CreateTablet for tablet 33e4a11e5f444b968f70470cf398354b (DEFAULT_TABLE table=test-table [id=5b296ca7c0194ee9bf61fd54cad9aa85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:21.667434 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e4a11e5f444b968f70470cf398354b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:21.667951 18602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33e4a11e5f444b968f70470cf398354b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:21.674647 18814 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Bootstrap starting.
I20260620 07:14:21.674647 18813 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Bootstrap starting.
I20260620 07:14:21.677243 18814 tablet_bootstrap.cc:654] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:21.677249 18813 tablet_bootstrap.cc:654] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:21.678052 18813 log.cc:824] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:21.678057 18814 log.cc:824] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:21.679895 18814 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: No bootstrap required, opened a new log
I20260620 07:14:21.679916 18813 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: No bootstrap required, opened a new log
I20260620 07:14:21.680124 18813 ts_tablet_manager.cc:1397] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:14:21.680124 18814 ts_tablet_manager.cc:1397] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:14:21.683575 18814 raft_consensus.cc:348] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.683573 18813 raft_consensus.cc:348] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.683820 18814 raft_consensus.cc:374] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:21.683852 18813 raft_consensus.cc:374] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:21.683880 18814 raft_consensus.cc:729] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3192261f5cc84ae28ebfec8ccbb61e0e, State: Initialized, Role: FOLLOWER
I20260620 07:14:21.683936 18813 raft_consensus.cc:729] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b445fa8bcaa14c5c8c91207fe930ea56, State: Initialized, Role: FOLLOWER
I20260620 07:14:21.684386 18814 consensus_queue.cc:260] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.684387 18813 consensus_queue.cc:260] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.685143 18800 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
I20260620 07:14:21.685323 18668 heartbeater.cc:503] Master 127.3.245.62:45645 was elected leader, sending a full tablet report...
I20260620 07:14:21.685537 18813 ts_tablet_manager.cc:1428] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 07:14:21.686383 18814 ts_tablet_manager.cc:1428] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260620 07:14:21.704864 18531 tablet.cc:2369] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:21.713690 18818 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:21.713896 18818 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.715131 18818 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533), a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425)
I20260620 07:14:21.719446 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.715341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40533 (local address 127.3.245.2:55525)
0620 07:14:21.715458 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.715477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.715580 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.715908 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.715913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.715962 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:21.716092 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.716099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.717117 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.717120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.718073 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.718083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.718912 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.718915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.719032 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.719059 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.719168 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.719296 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260620 07:14:21.719497 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.715433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:55525 (local address 127.3.245.3:40533)
0620 07:14:21.715560 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.715564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.715580 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:21.715718 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.715721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.715783 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.715871 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:21.716298 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.716996 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.718250 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.718777 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.718865 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.718921 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.719255 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.719347 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:14:21.719805 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.715432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37425 (local address 127.3.245.2:44109)
0620 07:14:21.715802 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.715817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:21.715904 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:21.716280 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:21.716283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:21.716304 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:21.716420 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.716427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.717493 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.717495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.718399 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:21.718409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.719303 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.719307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:21.719435 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.719458 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:14:21.719552 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:14:21.719627 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":132,"threads_started":1}
I20260620 07:14:21.719870 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.715665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:44109 (local address 127.3.245.1:37425)
0620 07:14:21.715859 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.715865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.715881 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:21.716016 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.716019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.716091 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.716196 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:21.716574 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.717355 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.718552 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.719155 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.719257 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.719437 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.719583 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.719672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:14:21.720271 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e4a11e5f444b968f70470cf398354b" candidate_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" is_pre_election: true
I20260620 07:14:21.720467 18484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e4a11e5f444b968f70470cf398354b" candidate_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cefb2543164df79070ee05a7aeb48f" is_pre_election: true
I20260620 07:14:21.720629 18754 raft_consensus.cc:2454] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3192261f5cc84ae28ebfec8ccbb61e0e in term 0.
I20260620 07:14:21.720904 18484 raft_consensus.cc:2454] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3192261f5cc84ae28ebfec8ccbb61e0e in term 0.
I20260620 07:14:21.721181 18555 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56; no voters:
I20260620 07:14:21.721542 18818 raft_consensus.cc:2781] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:21.721652 18818 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:21.721709 18818 raft_consensus.cc:3037] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:21.722967 18818 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.723376 18818 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [CANDIDATE]: Term 1 election: Requested vote from peers b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533), a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425)
I20260620 07:14:21.723869 18484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e4a11e5f444b968f70470cf398354b" candidate_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cefb2543164df79070ee05a7aeb48f"
I20260620 07:14:21.723879 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33e4a11e5f444b968f70470cf398354b" candidate_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
I20260620 07:14:21.724022 18484 raft_consensus.cc:3037] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:21.723996 18754 raft_consensus.cc:3037] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:21.725229 18484 raft_consensus.cc:2454] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3192261f5cc84ae28ebfec8ccbb61e0e in term 1.
I20260620 07:14:21.725402 18754 raft_consensus.cc:2454] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3192261f5cc84ae28ebfec8ccbb61e0e in term 1.
I20260620 07:14:21.725579 18556 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3192261f5cc84ae28ebfec8ccbb61e0e, a2cefb2543164df79070ee05a7aeb48f; no voters:
I20260620 07:14:21.725838 18818 raft_consensus.cc:2781] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:21.726086 18818 raft_consensus.cc:686] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 LEADER]: Becoming Leader. State: Replica: 3192261f5cc84ae28ebfec8ccbb61e0e, State: Running, Role: LEADER
I20260620 07:14:21.726468 18818 consensus_queue.cc:237] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:21.729820 18341 catalog_manager.cc:5495] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2). New cstate: current_term: 1 leader_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } health_report { overall_health: UNKNOWN } } }
I20260620 07:14:21.746524 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.742650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.3.245.2:39961)
0620 07:14:21.742828 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:21.742831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:21.742846 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:14:21.742923 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:21.742925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:21.742981 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:21.743073 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:21.743366 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.744013 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.745038 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:21.745281 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:21.745375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:21.745674 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:21.746122 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:21.746244 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:21.746274 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 07:14:21.746330 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:14:21.753114 18754 raft_consensus.cc:1261] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Refusing update from remote peer 3192261f5cc84ae28ebfec8ccbb61e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:21.753114 18484 raft_consensus.cc:1261] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Refusing update from remote peer 3192261f5cc84ae28ebfec8ccbb61e0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:21.753981 18818 consensus_queue.cc:1035] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:21.754269 18820 consensus_queue.cc:1035] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:21.760119 18823 mvcc.cc:205] Tried to move back new op lower bound from 7298824854532636672 to 7298824854434729984. Current Snapshot: MvccSnapshot[applied={T|T < 7298824854532636672}]
I20260620 07:14:21.762322 18825 mvcc.cc:205] Tried to move back new op lower bound from 7298824854532636672 to 7298824854434729984. Current Snapshot: MvccSnapshot[applied={T|T < 7298824854532636672}]
I20260620 07:14:21.762378 18824 mvcc.cc:205] Tried to move back new op lower bound from 7298824854532636672 to 7298824854434729984. Current Snapshot: MvccSnapshot[applied={T|T < 7298824854532636672 or (T in {7298824854532636672})}]
I20260620 07:14:21.770495 18341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:21.770696 18341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:21.773221 18341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Sending DeleteTablet for 3 replicas of tablet 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:21.773985 18464 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e4a11e5f444b968f70470cf398354b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:21 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:58944
I20260620 07:14:21.774019 18734 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e4a11e5f444b968f70470cf398354b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:21 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:47146
I20260620 07:14:21.774299 18602 tablet_service.cc:1514] Processing DeleteTablet for tablet 33e4a11e5f444b968f70470cf398354b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:21 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:37302
I20260620 07:14:21.774605 18833 tablet_replica.cc:330] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: stopping tablet replica
I20260620 07:14:21.774866 18834 tablet_replica.cc:330] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: stopping tablet replica
I20260620 07:14:21.774993 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18380
I20260620 07:14:21.775017 18833 raft_consensus.cc:2229] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:21.775254 18834 raft_consensus.cc:2229] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:21.775758 18833 raft_consensus.cc:2258] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:21.775753 18834 raft_consensus.cc:2258] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:21.777468 18833 ts_tablet_manager.cc:1905] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:21.777678 18834 ts_tablet_manager.cc:1905] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:21.780445 18834 ts_tablet_manager.cc:1918] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:21.780652 18834 log.cc:1196] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/33e4a11e5f444b968f70470cf398354b
I20260620 07:14:21.780661 18833 ts_tablet_manager.cc:1918] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:21.780759 18833 log.cc:1196] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/33e4a11e5f444b968f70470cf398354b
I20260620 07:14:21.781126 18833 ts_tablet_manager.cc:1939] T 33e4a11e5f444b968f70470cf398354b P b445fa8bcaa14c5c8c91207fe930ea56: Deleting consensus metadata
I20260620 07:14:21.781139 18834 ts_tablet_manager.cc:1939] T 33e4a11e5f444b968f70470cf398354b P 3192261f5cc84ae28ebfec8ccbb61e0e: Deleting consensus metadata
I20260620 07:14:21.782500 18326 catalog_manager.cc:4841] TS b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): tablet 33e4a11e5f444b968f70470cf398354b (table test-table [id=5b296ca7c0194ee9bf61fd54cad9aa85]) successfully deleted
I20260620 07:14:21.782500 18327 catalog_manager.cc:4841] TS 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): tablet 33e4a11e5f444b968f70470cf398354b (table test-table [id=5b296ca7c0194ee9bf61fd54cad9aa85]) successfully deleted
W20260620 07:14:21.783882 18327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37425 (error 108)
I20260620 07:14:21.784348 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18534
I20260620 07:14:21.785837 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:21.785579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37425 (local address 127.0.0.1:58950)
0620 07:14:21.785691 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:21.785765 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 07:14:21.786103 18327 catalog_manager.cc:4584] TS a2cefb2543164df79070ee05a7aeb48f (127.3.245.1:37425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33e4a11e5f444b968f70470cf398354b: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37425: connect: Connection refused (error 111)
I20260620 07:14:21.790448 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18671
I20260620 07:14:21.796012 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18309
I20260620 07:14:21.802930 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--webserver_interface=127.3.245.62
--webserver_port=36723
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:21.917975 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:21.918306 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:21.918401 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:21.922730 18836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:21.922837 18836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:21.922875 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:21.922914 18836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:21.922950 18836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:21.928679 18836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=36723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:21.929890 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:21.930958 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:21.939702 18841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.939684 18842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:21.939637 18844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:21.939965 18836 server_base.cc:1034] running on GCE node
I20260620 07:14:21.940824 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:21.941946 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:21.943145 18836 hybrid_clock.cc:648] HybridClock initialized: now 1781939661943127 us; error 27 us; skew 500 ppm
Jun 20 07:14:21 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:21.946041 18836 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:21.947000 18836 webserver.cc:460] Webserver started at http://127.3.245.62:36723/ using document root <none> and password file <none>
I20260620 07:14:21.947570 18836 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:21.947666 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:21.951785 18836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:21.954008 18851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:21.955327 18836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:14:21.955490 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "0bc7ff63f69d4b518aa8c240edd3064f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "6f059a5658f5dcc23918fb59d4d668bb"
server_key_iv: "79a87024134cd50e995304b74059223d"
server_key_version: "encryptionkey@0"
I20260620 07:14:21.956070 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:21.984041 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:21.987645 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:21.987869 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:21.995452 18836 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45645
I20260620 07:14:21.995770 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45645 every 8 connection(s)
I20260620 07:14:21.996593 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:22.000005 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18836
I20260620 07:14:22.000483 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:37425
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=39141
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:22.001101 18904 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:14:22.002761 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap starting.
I20260620 07:14:22.020774 18904 log.cc:824] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:22.031977 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:14:22.032392 18904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f: Bootstrap complete.
I20260620 07:14:22.037148 18904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:22.037823 18904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Initialized, Role: FOLLOWER
I20260620 07:14:22.038408 18904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:22.038585 18904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:22.038650 18904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:22.038774 18904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:14:22.040787 18904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:22.041119 18904 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc7ff63f69d4b518aa8c240edd3064f; no voters:
I20260620 07:14:22.041441 18904 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:14:22.041590 18908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:14:22.041842 18908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [term 3 LEADER]: Becoming Leader. State: Replica: 0bc7ff63f69d4b518aa8c240edd3064f, State: Running, Role: LEADER
I20260620 07:14:22.042150 18908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } }
I20260620 07:14:22.042707 18904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:22.043677 18910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc7ff63f69d4b518aa8c240edd3064f. Latest consensus state: current_term: 3 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:22.043831 18910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:22.044194 18917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:22.044169 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc7ff63f69d4b518aa8c240edd3064f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45645 } } }
I20260620 07:14:22.044255 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc7ff63f69d4b518aa8c240edd3064f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:22.049844 18917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:22.050516 18917 catalog_manager.cc:1259] Loaded cluster ID: 77f460744d104a2b836bbbd31949cdb1
I20260620 07:14:22.050578 18917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 07:14:22.051546 18917 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f362e53a9ad google::LogMessage::Fail() at ??:0
@ 0x7f362e53d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f362e53a4de google::LogMessage::Flush() at ??:0
@ 0x7f362e53ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3638e79fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3638e7ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3638e77579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3638ebb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f363078c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f363078cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f363078e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f363077d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3631dc16db start_thread at ??:0
@ 0x7f362d03861f clone at ??:0
W20260620 07:14:22.136270 18906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:22.136686 18906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:22.136776 18906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:22.143483 18906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:22.143571 18906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:22.143661 18906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:22.152287 18906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=39141
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:22.153942 18906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:22.155498 18906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:22.169449 18948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.169566 18906 server_base.cc:1034] running on GCE node
W20260620 07:14:22.169620 18945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.169463 18946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.170225 18906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:22.170917 18906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:22.172104 18906 hybrid_clock.cc:648] HybridClock initialized: now 1781939662172079 us; error 47 us; skew 500 ppm
Jun 20 07:14:22 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:22.175256 18906 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:22.176287 18906 webserver.cc:460] Webserver started at http://127.3.245.1:39141/ using document root <none> and password file <none>
I20260620 07:14:22.176972 18906 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:22.177070 18906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:22.181485 18906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:14:22.183943 18955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.185307 18906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:22.185443 18906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "a2cefb2543164df79070ee05a7aeb48f"
format_stamp: "Formatted at 2026-06-20 07:14:19 on dist-test-slave-ldfr"
server_key: "921782025376a791dfc260b8f90163da"
server_key_iv: "cb007b37c66c7bcb535d3e88db870619"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.186033 18906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:22.209301 18906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:22.213393 18906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:22.213645 18906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:22.214495 18906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:22.216177 18962 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:14:22.220417 18906 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:14:22.220494 18906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 07:14:22.220556 18906 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:14:22.222822 18906 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:14:22.222880 18906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:14:22.223223 18962 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Bootstrap starting.
I20260620 07:14:22.236109 18906 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37425
I20260620 07:14:22.236173 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37425 every 8 connection(s)
I20260620 07:14:22.237753 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 18906
I20260620 07:14:22.238193 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:39961
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=33163
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:14:22.239861 18906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:14:22.243454 18962 log.cc:824] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:22.250979 18962 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:14:22.251364 18962 tablet_bootstrap.cc:492] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Bootstrap complete.
I20260620 07:14:22.252483 18962 ts_tablet_manager.cc:1397] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.015s
I20260620 07:14:22.255002 18962 raft_consensus.cc:348] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:22.255426 18962 raft_consensus.cc:729] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cefb2543164df79070ee05a7aeb48f, State: Initialized, Role: FOLLOWER
I20260620 07:14:22.255918 18962 consensus_queue.cc:260] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:22.256920 18962 ts_tablet_manager.cc:1428] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f: Time spent starting tablet: real 0.004s user 0.006s sys 0.002s
W20260620 07:14:22.337287 19073 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:14:22.242237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:53911)
0620 07:14:22.248784 (+ 6547us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.248845 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:22.249794 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:22.336362 (+ 86568us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1972,"thread_start_us":148,"threads_started":1}
W20260620 07:14:22.337942 19071 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45645 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45645: Client connection negotiation failed: client connection to 127.3.245.62:45645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:14:22.338753 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.338480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:53441)
0620 07:14:22.338612 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.338689 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 07:14:22.339555 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.339263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:38749)
0620 07:14:22.339422 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.339487 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:14:22.339838 19071 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:14:22.379875 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:22.380167 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:22.380254 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:22.385025 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:22.385098 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:22.385213 19070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:22.391355 19070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:39961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=33163
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:22.392516 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:22.393693 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:22.404237 19081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.404304 19080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.404305 19083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.404547 19070 server_base.cc:1034] running on GCE node
I20260620 07:14:22.405118 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:22.405843 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:22.407068 19070 hybrid_clock.cc:648] HybridClock initialized: now 1781939662407055 us; error 49 us; skew 500 ppm
Jun 20 07:14:22 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:22.410235 19070 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:22.411274 19070 webserver.cc:460] Webserver started at http://127.3.245.2:33163/ using document root <none> and password file <none>
I20260620 07:14:22.411947 19070 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:22.412056 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:22.416450 19070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:22.418838 19090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.420172 19070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:22.420320 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "3192261f5cc84ae28ebfec8ccbb61e0e"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "3bc93a3a0cdef469863e9d4026e20eed"
server_key_iv: "e3ae7f3c74751f569cc4c5d1a60dab32"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.420955 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:22.432036 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:22.435211 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:22.435413 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:22.436043 19070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:22.436949 19070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:22.437021 19070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.437093 19070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:22.437136 19070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.447093 19070 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:39961
I20260620 07:14:22.447139 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:39961 every 8 connection(s)
I20260620 07:14:22.448158 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:22.452273 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.450574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:60431)
0620 07:14:22.451344 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.451458 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":124,"threads_started":1}
W20260620 07:14:22.452903 19204 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45645 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45645: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
I20260620 07:14:22.453589 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.453330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:48047)
0620 07:14:22.453462 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.453523 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:14:22.454326 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.454094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:45023)
0620 07:14:22.454220 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.454275 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:14:22.454500 19204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:14:22.456944 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19070
I20260620 07:14:22.457334 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:40533
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:45645
--webserver_port=32843
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:22.569654 19207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:22.569983 19207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:22.570079 19207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:22.574259 19207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:22.574349 19207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:22.574441 19207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:22.579722 19207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:40533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=32843
--tserver_master_addrs=127.3.245.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:22.580863 19207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:22.581903 19207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:14:22.592234 19207 server_base.cc:1034] running on GCE node
W20260620 07:14:22.592180 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.592180 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:22.592209 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:22.593067 19207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:22.593719 19207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:22.594905 19207 hybrid_clock.cc:648] HybridClock initialized: now 1781939662594800 us; error 118 us; skew 500 ppm
Jun 20 07:14:22 dist-test-slave-ldfr krb5kdc[17760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:22.597824 19207 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:22.598790 19207 webserver.cc:460] Webserver started at http://127.3.245.3:32843/ using document root <none> and password file <none>
I20260620 07:14:22.599395 19207 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:22.599493 19207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:22.603551 19207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:14:22.605803 19222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.607084 19207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:14:22.607219 19207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b445fa8bcaa14c5c8c91207fe930ea56"
format_stamp: "Formatted at 2026-06-20 07:14:20 on dist-test-slave-ldfr"
server_key: "d3d85528e46f8926d981ccd426dcffc2"
server_key_iv: "79f777039ef1b6d4952ef5f4bb868548"
server_key_version: "encryptionkey@0"
I20260620 07:14:22.607816 19207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:22.640558 19207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:22.643744 19207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:22.643955 19207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:22.644546 19207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:22.645391 19207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:22.645473 19207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.645548 19207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:22.645591 19207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:22.654871 19207 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:40533
I20260620 07:14:22.654897 19335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:40533 every 8 connection(s)
I20260620 07:14:22.655851 19207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:22.659721 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.657986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:51037)
0620 07:14:22.658656 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.658782 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":117,"threads_started":1}
W20260620 07:14:22.660197 19336 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45645 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45645: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
I20260620 07:14:22.660820 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.660535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:45111)
0620 07:14:22.660663 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.660765 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:14:22.661523 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:22.661313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:38341)
0620 07:14:22.661420 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:22.661471 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:14:22.661808 19336 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:14:22.664242 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19207
I20260620 07:14:23.340965 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.340444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:57837)
0620 07:14:23.340779 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.340844 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":76,"threads_started":1}
I20260620 07:14:23.455574 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.455137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:49981)
0620 07:14:23.455404 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.455467 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260620 07:14:23.478780 19342 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:23.479001 19342 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:23.480171 19342 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
Jun 20 07:14:23 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.2@KRBTEST.COM
Jun 20 07:14:23 dist-test-slave-ldfr krb5kdc[17760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939662, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.3@KRBTEST.COM
I20260620 07:14:23.494076 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.480323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:39961 (local address 127.3.245.1:36717)
0620 07:14:23.480490 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.480502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.480589 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.482501 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.482507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.483056 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:23.483541 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.483556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.484997 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.484999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.486017 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.486026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.486757 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.486991 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.487690 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:23.487702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:23.489246 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:23.491693 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:23.491699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:23.491712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:23.491956 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:23.492306 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:23.492312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:23.492459 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:23.493137 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:23.493143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:23.493366 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.493548 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.493879 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":190,"spinlock_wait_cycles":12672}
I20260620 07:14:23.494091 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.480545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:36717 (local address 127.3.245.2:39961)
0620 07:14:23.480947 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.480971 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.481019 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:14:23.481708 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.481714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.482133 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.482387 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.483730 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.484854 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486144 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.486634 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486794 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.489499 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.489507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.489525 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:23.489565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.491558 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492074 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:23.492080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:23.492084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:23.492204 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492556 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:23.492561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:23.492563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:23.493007 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.493142 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.493675 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.493930 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":156,"threads_started":1}
I20260620 07:14:23.494132 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.481150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:57583 (local address 127.3.245.3:40533)
0620 07:14:23.481408 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:23.481428 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:23.481461 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:14:23.482277 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:23.482284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:23.482706 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:23.483005 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:23.483718 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.484838 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486181 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.486660 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486836 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.489860 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:23.489864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:23.489875 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:23.489925 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:23.491767 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492202 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:23.492208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:23.492218 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:23.492327 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492650 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:23.492653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:23.492655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:23.493028 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:23.493167 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:23.493671 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 07:14:23.493948 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260620 07:14:23.494122 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.480521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40533 (local address 127.3.245.1:57583)
0620 07:14:23.480882 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.480895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:23.480978 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:23.482958 (+ 1980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:23.482960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:23.483133 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:23.483541 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.483556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.484986 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.484990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.486018 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:23.486028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:23.486803 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:23.486807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:23.486987 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:23.487523 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:23.487539 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:23.489609 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:23.491885 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:23.491890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:23.491893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:23.492089 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:23.492424 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:23.492429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:23.492431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:23.492534 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:23.493136 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:23.493146 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:23.493371 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 07:14:23.493549 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 07:14:23.493964 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":268,"spinlock_wait_cycles":27264,"thread_start_us":192,"threads_started":1}
W20260620 07:14:23.496515 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:23.496701 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:23.496812 18956 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:23.497089 19342 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:23.662766 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:23.662372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:42221)
0620 07:14:23.662614 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:23.662668 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260620 07:14:24.342368 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:50523)
0620 07:14:24.342190 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.342261 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":83,"threads_started":1}
I20260620 07:14:24.456943 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.456415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:56405)
0620 07:14:24.456786 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.456850 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":91,"threads_started":1}
I20260620 07:14:24.663946 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:24.663524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:60047)
0620 07:14:24.663777 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:24.663834 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 07:14:25.298595 19350 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:25.298734 19350 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:25.299219 19350 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
W20260620 07:14:25.300145 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:25.300318 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:25.300395 18957 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:25.300523 19350 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:25.343761 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:25.343326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:43959)
0620 07:14:25.343603 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:25.343663 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260620 07:14:25.458308 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:25.457853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:40887)
0620 07:14:25.458129 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:25.458201 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260620 07:14:25.665331 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:25.664899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:46509)
0620 07:14:25.665174 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:25.665233 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260620 07:14:26.345153 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:26.344582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:42243)
0620 07:14:26.344967 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:26.345029 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":91,"threads_started":1}
I20260620 07:14:26.459523 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:26.459108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:41479)
0620 07:14:26.459375 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:26.459432 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260620 07:14:26.666564 19356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:26.666126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:45565)
0620 07:14:26.666397 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:26.666456 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260620 07:14:27.346729 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:27.346225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:50861)
0620 07:14:27.346568 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:27.346633 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260620 07:14:27.461001 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:27.460465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:36427)
0620 07:14:27.460851 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:27.460909 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":133,"threads_started":1}
I20260620 07:14:27.667831 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:27.667424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:52275)
0620 07:14:27.667691 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:27.667742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260620 07:14:28.345018 19360 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:28.345176 19360 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:28.345705 19360 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
W20260620 07:14:28.346598 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:28.346748 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:28.346822 18957 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:28.346989 19360 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:28.347882 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:28.347400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:60983)
0620 07:14:28.347732 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:28.347794 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":85,"threads_started":1}
I20260620 07:14:28.462446 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:28.462012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:42487)
0620 07:14:28.462293 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:28.462353 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":119,"threads_started":1}
I20260620 07:14:28.669376 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:28.668739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:55449)
0620 07:14:28.669132 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:28.669229 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":111,"threads_started":1}
I20260620 07:14:29.349148 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:29.348719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:55135)
0620 07:14:29.348990 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:29.349051 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
I20260620 07:14:29.463902 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:29.463370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:47175)
0620 07:14:29.463733 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:29.463805 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":104,"threads_started":1}
I20260620 07:14:29.670650 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:29.670200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:50069)
0620 07:14:29.670487 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:29.670546 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":72,"threads_started":1}
I20260620 07:14:30.350651 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:30.350157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:35717)
0620 07:14:30.350465 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:30.350544 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260620 07:14:30.465346 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:30.464893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:53861)
0620 07:14:30.465166 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:30.465243 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":122,"threads_started":1}
I20260620 07:14:30.671862 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:30.671434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:42299)
0620 07:14:30.671706 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:30.671766 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260620 07:14:31.035038 19370 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:31.035208 19370 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:31.035583 19370 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
W20260620 07:14:31.036487 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:31.036698 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:31.036775 18957 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:31.036900 19370 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:31.351907 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:31.351487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:41045)
0620 07:14:31.351761 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:31.351814 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260620 07:14:31.466741 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:31.466204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:38771)
0620 07:14:31.466514 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:31.466610 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260620 07:14:31.673069 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:31.672684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:41075)
0620 07:14:31.672916 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:31.672970 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 07:14:32.353268 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:32.352872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:32777)
0620 07:14:32.353112 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:32.353165 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260620 07:14:32.468284 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:32.467721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:35967)
0620 07:14:32.468108 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:32.468176 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":91,"threads_started":1}
I20260620 07:14:32.674319 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:32.673911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:36147)
0620 07:14:32.674157 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:32.674217 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260620 07:14:33.354544 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:33.354126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:60609)
0620 07:14:33.354383 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:33.354436 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260620 07:14:33.469817 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:33.469368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:44467)
0620 07:14:33.469645 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:33.469714 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260620 07:14:33.675529 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:33.675070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:56415)
0620 07:14:33.675365 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:33.675428 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":99,"threads_started":1}
I20260620 07:14:33.851701 19380 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:33.851852 19380 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:33.852274 19380 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
W20260620 07:14:33.853066 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:33.853173 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:33.853293 18957 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:33.853437 19380 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:34.355813 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:34.355441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:33873)
0620 07:14:34.355669 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:34.355718 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260620 07:14:34.471104 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:34.470641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:36935)
0620 07:14:34.470946 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:34.471010 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260620 07:14:34.676878 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:34.676408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:44531)
0620 07:14:34.676723 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:34.676783 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260620 07:14:35.357070 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:35.356533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:60675)
0620 07:14:35.356912 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:35.356967 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":131,"threads_started":1}
I20260620 07:14:35.472445 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:35.472018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:57755)
0620 07:14:35.472291 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:35.472349 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260620 07:14:35.678083 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:35.677656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:35039)
0620 07:14:35.677914 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:35.677974 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260620 07:14:36.358294 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:36.357867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:51171)
0620 07:14:36.358120 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:36.358193 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260620 07:14:36.396353 19388 raft_consensus.cc:482] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:36.396534 19388 raft_consensus.cc:504] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3192261f5cc84ae28ebfec8ccbb61e0e" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 39961 } } peers { permanent_uuid: "b445fa8bcaa14c5c8c91207fe930ea56" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40533 } } peers { permanent_uuid: "a2cefb2543164df79070ee05a7aeb48f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37425 } }
I20260620 07:14:36.397200 19388 leader_election.cc:288] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961), b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533)
W20260620 07:14:36.398044 18957 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3192261f5cc84ae28ebfec8ccbb61e0e (127.3.245.2:39961): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
W20260620 07:14:36.398166 18956 leader_election.cc:341] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b445fa8bcaa14c5c8c91207fe930ea56 (127.3.245.3:40533): Not found: Tablet not found: 33e4a11e5f444b968f70470cf398354b
I20260620 07:14:36.398275 18956 leader_election.cc:302] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2cefb2543164df79070ee05a7aeb48f; no voters: 3192261f5cc84ae28ebfec8ccbb61e0e, b445fa8bcaa14c5c8c91207fe930ea56
I20260620 07:14:36.398475 19388 raft_consensus.cc:2726] T 33e4a11e5f444b968f70470cf398354b P a2cefb2543164df79070ee05a7aeb48f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:14:36.473865 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:36.473411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:46059)
0620 07:14:36.473703 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:36.473769 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260620 07:14:36.679340 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:36.678932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:40965)
0620 07:14:36.679182 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:36.679239 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260620 07:14:37.359759 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:37.359333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.1:37749)
0620 07:14:37.359612 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:37.359668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260620 07:14:37.475353 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:37.474904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.2:33607)
0620 07:14:37.475195 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:37.475256 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260620 07:14:37.675437 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 18906
I20260620 07:14:37.680814 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:37.680354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45645 (local address 127.3.245.3:44725)
0620 07:14:37.680592 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:37.680715 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260620 07:14:37.682848 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19070
I20260620 07:14:37.688676 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19207
2026-06-20T07:14:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19533 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:14:37 dist-test-slave-ldfr krb5kdc[19396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:14:37 dist-test-slave-ldfr krb5kdc[19396](info): set up 2 sockets
Jun 20 07:14:37 dist-test-slave-ldfr krb5kdc[19396](info): commencing operation
krb5kdc: starting...
W20260620 07:14:38.756999 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:14:38 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:14:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:14:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:14:38.884109 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:39.001626 19409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:39.001965 19409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:39.002068 19409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:39.006312 19409 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:14:39.006444 19409 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:14:39.006476 19409 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:14:39.006515 19409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:14:39.006552 19409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:39.006597 19409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:39.006635 19409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:14:39.006675 19409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:14:39.012451 19409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:39.013674 19409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:39.014761 19409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:39.026710 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.026713 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.026708 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.027101 19409 server_base.cc:1034] running on GCE node
I20260620 07:14:39.027909 19409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:39.029587 19409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:39.030808 19409 hybrid_clock.cc:648] HybridClock initialized: now 1781939679030779 us; error 41 us; skew 500 ppm
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:39.035151 19409 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:14:39.036460 19409 webserver.cc:460] Webserver started at http://127.3.245.62:44509/ using document root <none> and password file <none>
I20260620 07:14:39.037295 19409 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:39.037396 19409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:39.037665 19409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:39.040424 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "131b583e25474a1b97eb4d2fd9a3511f"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "91918822b60d24b89cc4dcc7a337a248"
server_key_iv: "f8b9196b549ec7444be0bf1193367897"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.041209 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "131b583e25474a1b97eb4d2fd9a3511f"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "91918822b60d24b89cc4dcc7a337a248"
server_key_iv: "f8b9196b549ec7444be0bf1193367897"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.047154 19409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 07:14:39.050786 19424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.052760 19409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:14:39.052955 19409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "131b583e25474a1b97eb4d2fd9a3511f"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "91918822b60d24b89cc4dcc7a337a248"
server_key_iv: "f8b9196b549ec7444be0bf1193367897"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.053124 19409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:39.073791 19409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:39.077703 19409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:39.077906 19409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:39.086440 19409 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45989
I20260620 07:14:39.086741 19476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45989 every 8 connection(s)
I20260620 07:14:39.087663 19409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:14:39.091048 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:39.091619 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19409
I20260620 07:14:39.091794 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:14:39.092105 4052 external_mini_cluster.cc:1428] Setting key bbbba2089c270e92b6eef6ed891d8862
I20260620 07:14:39.098584 19477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap starting.
I20260620 07:14:39.101672 19477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:39.102626 19477 log.cc:824] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:39.105119 19477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: No bootstrap required, opened a new log
I20260620 07:14:39.109066 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.093847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.3.245.62:45989)
0620 07:14:39.094317 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.094335 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.094373 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:14:39.095081 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.095099 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.095495 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.095805 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.096850 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.098124 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.099443 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.100242 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.100450 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.103537 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.103551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.103571 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:39.103604 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.105581 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.106229 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.106235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.106240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.106378 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.106703 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.106706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.106707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.107208 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.107377 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.107798 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.108180 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":142,"threads_started":1}
I20260620 07:14:39.109517 19477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:14:39.109797 19477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:39.109890 19477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Initialized, Role: FOLLOWER
I20260620 07:14:39.110417 19477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:14:39.110584 19477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:14:39.110672 19477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:14:39.110792 19477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:39.111824 19477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:14:39.112207 19477 leader_election.cc:302] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 131b583e25474a1b97eb4d2fd9a3511f; no voters:
I20260620 07:14:39.112562 19477 leader_election.cc:288] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:14:39.112898 19482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:39.113119 19482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 LEADER]: Becoming Leader. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Running, Role: LEADER
I20260620 07:14:39.113472 19482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:14:39.113977 19477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:14:39.115321 19483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:14:39.115456 19483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
I20260620 07:14:39.115177 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 131b583e25474a1b97eb4d2fd9a3511f. Latest consensus state: current_term: 1 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:14:39.116070 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
W20260620 07:14:39.118700 19497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:14:39.118775 19497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:14:39.118856 19492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:14:39.119920 19492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:14:39.126801 19492 catalog_manager.cc:1347] Generated new cluster ID: c08effe862454fb1979114d578fbc447
I20260620 07:14:39.126894 19492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:14:39.184167 19492 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:14:39.185673 19492 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:14:39.192330 19492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Generated new TSK 0
I20260620 07:14:39.193109 19492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:14:39.229527 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:39.345789 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:39.346145 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:39.346241 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:39.350507 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:39.350610 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:39.350704 19503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:14:39.356150 19503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:39.357369 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:39.358420 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:39.368494 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.368513 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.368533 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.368842 19503 server_base.cc:1034] running on GCE node
I20260620 07:14:39.369316 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:39.369984 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:39.371184 19503 hybrid_clock.cc:648] HybridClock initialized: now 1781939679371164 us; error 31 us; skew 500 ppm
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:39.374248 19503 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:14:39.375139 19503 webserver.cc:460] Webserver started at http://127.3.245.1:34765/ using document root <none> and password file <none>
I20260620 07:14:39.375777 19503 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:39.375876 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:39.376116 19503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:39.378306 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "20c635026273488a943c9e600acafb42"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "0c7db101b0be440ae027dd42eed6cb9e"
server_key_iv: "a5eaf4aab487dceb2c725dc97a15e6ca"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.378881 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "20c635026273488a943c9e600acafb42"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "0c7db101b0be440ae027dd42eed6cb9e"
server_key_iv: "a5eaf4aab487dceb2c725dc97a15e6ca"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.382884 19503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:14:39.385555 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.386895 19503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:39.387091 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "20c635026273488a943c9e600acafb42"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "0c7db101b0be440ae027dd42eed6cb9e"
server_key_iv: "a5eaf4aab487dceb2c725dc97a15e6ca"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.387234 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:39.409250 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:39.412761 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:39.412998 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:39.413702 19503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:39.414769 19503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:39.414846 19503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.414920 19503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:39.415009 19503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.425838 19503 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37045
I20260620 07:14:39.425858 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37045 every 8 connection(s)
I20260620 07:14:39.426851 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:39.437073 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19503
I20260620 07:14:39.437227 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:14:39.437537 4052 external_mini_cluster.cc:1428] Setting key 26579b2b9a946e20ca0df768c4fce1b4
I20260620 07:14:39.441867 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.428980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:57617 (local address 127.3.245.62:45989)
0620 07:14:39.429187 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.429192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.430301 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 07:14:39.431209 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.431212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.431263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.431351 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.432657 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.433266 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.434556 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.434809 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.434904 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.437877 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.437880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.437882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:39.437908 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.439465 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.440115 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.440118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.440119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.440196 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.440637 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.440642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.440646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.440905 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.441014 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.441528 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.441685 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 07:14:39.442885 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.429422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:57617)
0620 07:14:39.430140 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.430183 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.430954 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.431507 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.431513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.431933 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:39.432424 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.432437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.433406 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.433409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.434406 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.434416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.434986 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.434989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.435209 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.435916 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:39.435934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:39.437677 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:39.439584 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.439593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.439622 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.440009 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.440290 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.440296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.440301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.440501 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.441016 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:39.441022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:39.441287 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.441451 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.441781 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":134,"threads_started":1}
I20260620 07:14:39.444307 19632 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:14:39.444567 19632 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:39.445106 19632 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:14:39.446820 19441 ts_manager.cc:194] Registered new tserver with Master: 20c635026273488a943c9e600acafb42 (127.3.245.1:37045)
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:14:39.448372 19441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:57617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:14:39.468585 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:39.582342 19637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:39.582633 19637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:39.582679 19637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:39.586845 19637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:39.586915 19637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:39.586977 19637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:14:39.592386 19637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:39.593526 19637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:39.594674 19637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:39.605057 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.605057 19645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.605195 19637 server_base.cc:1034] running on GCE node
W20260620 07:14:39.605444 19643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.605979 19637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:39.606695 19637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:39.607877 19637 hybrid_clock.cc:648] HybridClock initialized: now 1781939679607857 us; error 32 us; skew 500 ppm
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:39.610956 19637 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:14:39.612039 19637 webserver.cc:460] Webserver started at http://127.3.245.2:41245/ using document root <none> and password file <none>
I20260620 07:14:39.612738 19637 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:39.612810 19637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:39.613076 19637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:39.615501 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "a845699406eb4de3a8438a6ee4fb6969"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "66754f2e5378ef4af857bbec63926759"
server_key_iv: "129a351fa23ca4376f25f9aef22ba852"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.616174 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "a845699406eb4de3a8438a6ee4fb6969"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "66754f2e5378ef4af857bbec63926759"
server_key_iv: "129a351fa23ca4376f25f9aef22ba852"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.620944 19637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260620 07:14:39.623899 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.625344 19637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:14:39.625469 19637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "a845699406eb4de3a8438a6ee4fb6969"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "66754f2e5378ef4af857bbec63926759"
server_key_iv: "129a351fa23ca4376f25f9aef22ba852"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.625587 19637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:39.645471 19637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:39.648841 19637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:39.649077 19637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:39.649705 19637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:39.650565 19637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:39.650619 19637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.650689 19637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:39.650727 19637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.660039 19637 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44893
I20260620 07:14:39.660058 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44893 every 8 connection(s)
I20260620 07:14:39.661029 19637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:14:39.665488 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19637
I20260620 07:14:39.665616 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:14:39.665907 4052 external_mini_cluster.cc:1428] Setting key 4c5f65047952c560d27d91c649b84d73
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:14:39.677395 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.662903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:41461 (local address 127.3.245.62:45989)
0620 07:14:39.663095 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.663099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.664167 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 07:14:39.665053 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.665056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.665106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.665192 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.666557 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.667397 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.669263 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.669666 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.669795 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.672964 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.672968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.672970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:39.672997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.674829 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.675464 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.675468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.675469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.675555 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.675951 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.675954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.675956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.676246 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.676371 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.677044 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.677202 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 07:14:39.678340 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.663297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:41461)
0620 07:14:39.664009 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.664052 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.664841 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.665346 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.665353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.665830 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:39.666362 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.666377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.667541 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.667545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.669099 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.669114 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.669832 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.669835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.669983 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.670657 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:39.670688 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:39.672757 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:39.674984 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.674991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.675018 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.675326 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.675673 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.675681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.675683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.675837 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.676367 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:39.676376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:39.676770 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.676980 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.677361 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:14:39.679780 19766 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:14:39.680055 19766 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:39.680666 19766 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:14:39.681952 19441 ts_manager.cc:194] Registered new tserver with Master: a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
I20260620 07:14:39.682672 19441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:41461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:14:39.695858 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:14:39.807391 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:14:39.807802 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:14:39.807881 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:14:39.812036 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:14:39.812135 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:14:39.812251 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:14:39.817924 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:14:39.819072 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:14:39.820137 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:14:39.830111 19777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.830133 19779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:14:39.830118 19776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:14:39.830394 19771 server_base.cc:1034] running on GCE node
I20260620 07:14:39.830857 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:14:39.831480 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:14:39.832705 19771 hybrid_clock.cc:648] HybridClock initialized: now 1781939679832670 us; error 46 us; skew 500 ppm
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:14:39.835376 19771 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:14:39.836299 19771 webserver.cc:460] Webserver started at http://127.3.245.3:35213/ using document root <none> and password file <none>
I20260620 07:14:39.836964 19771 fs_manager.cc:359] Metadata directory not provided
I20260620 07:14:39.837055 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:14:39.837308 19771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:14:39.839349 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "b2f87febcbf54671bbbbc54c94a7518e"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "8ac6575d47666954bc5e9ea5ddbb75c3"
server_key_iv: "0bf1ae32bea4c9dead955132a314a3ce"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.839915 19771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "b2f87febcbf54671bbbbc54c94a7518e"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "8ac6575d47666954bc5e9ea5ddbb75c3"
server_key_iv: "0bf1ae32bea4c9dead955132a314a3ce"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.843608 19771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:14:39.846119 19786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.847435 19771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:14:39.847582 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b2f87febcbf54671bbbbc54c94a7518e"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "8ac6575d47666954bc5e9ea5ddbb75c3"
server_key_iv: "0bf1ae32bea4c9dead955132a314a3ce"
server_key_version: "encryptionkey@0"
I20260620 07:14:39.847718 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:14:39.865899 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:14:39.868671 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:14:39.868873 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:14:39.869460 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:14:39.870270 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:14:39.870338 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.870414 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:14:39.870456 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:14:39.879246 19771 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:45223
I20260620 07:14:39.879282 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:45223 every 8 connection(s)
I20260620 07:14:39.880146 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:14:39.882020 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19771
I20260620 07:14:39.882138 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:14:39.882432 4052 external_mini_cluster.cc:1428] Setting key a0ec7d776d4c437e9674b48ff7915fe9
Jun 20 07:14:39 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939679, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:14:39.894443 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.882147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:43641 (local address 127.3.245.62:45989)
0620 07:14:39.882303 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.882307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.883622 (+ 1315us) server_negotiation.cc:407] Connection header received
0620 07:14:39.884734 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.884737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.884791 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.884867 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.886019 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.886552 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.887701 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.887940 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.888053 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.890766 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.890770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.890773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:39.890826 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.892245 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.892900 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.892908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.892909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.892984 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.893342 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.893345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.893347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.893527 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.893635 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.894152 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.894275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:14:39.895259 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.882643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:43641)
0620 07:14:39.883477 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.883526 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.884481 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.885034 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.885041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.885397 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:14:39.885870 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.885882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.886691 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.886694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.887574 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.887583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.888097 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.888102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.888270 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.888974 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:14:39.888992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:14:39.890579 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:14:39.892364 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.892370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.892385 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.892787 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.893088 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:14:39.893093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:14:39.893097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:14:39.893241 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:14:39.893643 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:14:39.893651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:14:39.893902 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.894094 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.894422 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":168,"threads_started":1}
I20260620 07:14:39.896464 19900 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:14:39.896747 19900 heartbeater.cc:464] Registering TS with master...
I20260620 07:14:39.897267 19900 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:14:39.898379 19441 ts_manager.cc:194] Registered new tserver with Master: b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223)
I20260620 07:14:39.899021 19441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:43641
I20260620 07:14:39.909196 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:14:39.919467 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.911303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.3.245.62:45989)
0620 07:14:39.911477 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.911480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.911646 (+ 166us) server_negotiation.cc:407] Connection header received
0620 07:14:39.911691 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.911693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.911737 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.911857 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:14:39.912520 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.913133 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.914083 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.914429 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.914533 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.916116 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:14:39.916119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:14:39.916121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:14:39.916146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:14:39.917761 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.918219 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.918222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.918223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.918299 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:14:39.918562 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:14:39.918564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:14:39.918565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:14:39.918802 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:14:39.918911 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.919154 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.919253 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:14:39.922591 19441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:14:39.925889 19441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:14:39.940928 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.935605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.3.245.2:44893)
0620 07:14:39.935953 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.935968 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.935991 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:14:39.936067 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.936074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.936319 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.936876 (+ 557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:39.937003 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.937864 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.939341 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.940192 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.940317 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.940474 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.940656 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.940762 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":166,"threads_started":1}
I20260620 07:14:39.940917 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.934976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44893 (local address 127.0.0.1:51238)
0620 07:14:39.935240 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.935255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.935345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.936512 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.936518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.936542 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:39.936739 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.936747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.938005 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.938008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.939227 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.939237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.940316 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.940319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.940425 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.940463 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.940594 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.940771 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260620 07:14:39.940958 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.934777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45223 (local address 127.0.0.1:32812)
0620 07:14:39.935146 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.935200 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.935335 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.936732 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.936735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.936754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:39.936863 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.936870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.938525 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.938528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.939531 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.939539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.940468 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.940471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.940564 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.940590 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.940661 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.940797 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260620 07:14:39.941715 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.935532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37045 (local address 127.0.0.1:51900)
0620 07:14:39.936534 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:39.936547 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:39.936656 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:39.937100 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:39.937103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:39.937119 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:39.937208 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.937214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.938764 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.938769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.939921 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:39.939935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.941254 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.941258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:39.941374 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.941403 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:14:39.941504 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 07:14:39.941577 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":109,"threads_started":1}
I20260620 07:14:39.941936 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.935696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.3.245.1:37045)
0620 07:14:39.936309 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.936323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.936671 (+ 348us) server_negotiation.cc:407] Connection header received
0620 07:14:39.936754 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.936759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.936951 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.937092 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:39.937321 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.938432 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.940061 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.940929 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.941036 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.941190 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.941543 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.941639 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":117,"threads_started":1}
I20260620 07:14:39.942148 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:39.935869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.3.245.3:45223)
0620 07:14:39.936211 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:39.936222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:39.936247 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:14:39.936323 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:39.936329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:39.936526 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:39.937014 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:39.937187 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.938377 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.939644 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:39.940331 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:39.941580 (+ 1249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:39.941716 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:39.941782 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 07:14:39.942007 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":159,"threads_started":1}
I20260620 07:14:39.944407 19566 tablet_service.cc:1467] Processing CreateTablet for tablet bfe804a0922f4ff883878c780de2630d (DEFAULT_TABLE table=test-table [id=e04e9b579de944ff9007874a670c3972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:39.944414 19700 tablet_service.cc:1467] Processing CreateTablet for tablet bfe804a0922f4ff883878c780de2630d (DEFAULT_TABLE table=test-table [id=e04e9b579de944ff9007874a670c3972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:39.944703 19834 tablet_service.cc:1467] Processing CreateTablet for tablet bfe804a0922f4ff883878c780de2630d (DEFAULT_TABLE table=test-table [id=e04e9b579de944ff9007874a670c3972]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:14:39.945509 19566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe804a0922f4ff883878c780de2630d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:39.945533 19834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe804a0922f4ff883878c780de2630d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:39.945530 19700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe804a0922f4ff883878c780de2630d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:14:39.951884 19917 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Bootstrap starting.
I20260620 07:14:39.952997 19918 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Bootstrap starting.
I20260620 07:14:39.953819 19919 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Bootstrap starting.
I20260620 07:14:39.955024 19917 tablet_bootstrap.cc:654] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:39.955476 19918 tablet_bootstrap.cc:654] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:39.956014 19917 log.cc:824] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:39.956067 19919 tablet_bootstrap.cc:654] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Neither blocks nor log segments found. Creating new log.
I20260620 07:14:39.956248 19918 log.cc:824] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:39.956887 19919 log.cc:824] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:14:39.958252 19917 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: No bootstrap required, opened a new log
I20260620 07:14:39.958496 19917 ts_tablet_manager.cc:1397] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:14:39.958706 19919 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: No bootstrap required, opened a new log
I20260620 07:14:39.958897 19919 ts_tablet_manager.cc:1397] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:14:39.958988 19918 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: No bootstrap required, opened a new log
I20260620 07:14:39.959197 19918 ts_tablet_manager.cc:1397] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:14:39.962245 19919 raft_consensus.cc:348] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.962509 19919 raft_consensus.cc:374] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:39.962564 19919 raft_consensus.cc:729] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f87febcbf54671bbbbc54c94a7518e, State: Initialized, Role: FOLLOWER
I20260620 07:14:39.963018 19919 consensus_queue.cc:260] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.963153 19917 raft_consensus.cc:348] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.963153 19918 raft_consensus.cc:348] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.963433 19918 raft_consensus.cc:374] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:39.963433 19917 raft_consensus.cc:374] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:14:39.963497 19918 raft_consensus.cc:729] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845699406eb4de3a8438a6ee4fb6969, State: Initialized, Role: FOLLOWER
I20260620 07:14:39.963497 19917 raft_consensus.cc:729] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c635026273488a943c9e600acafb42, State: Initialized, Role: FOLLOWER
I20260620 07:14:39.963858 19900 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
I20260620 07:14:39.963979 19918 consensus_queue.cc:260] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.963979 19917 consensus_queue.cc:260] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:39.964632 19766 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
I20260620 07:14:39.964920 19632 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
I20260620 07:14:39.964993 19917 ts_tablet_manager.cc:1428] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:14:39.965929 19918 ts_tablet_manager.cc:1428] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:14:39.966701 19919 ts_tablet_manager.cc:1428] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:14:40.088455 19924 raft_consensus.cc:482] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:14:40.088728 19924 raft_consensus.cc:504] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:40.090034 19924 leader_election.cc:288] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), 20c635026273488a943c9e600acafb42 (127.3.245.1:37045)
I20260620 07:14:40.094544 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.090488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37045 (local address 127.3.245.2:47347)
0620 07:14:40.090878 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:40.090893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:40.090980 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:40.091214 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:40.091217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:40.091231 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:40.091329 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:40.091334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.092285 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.092289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:40.093212 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:40.093221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.094106 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.094109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:40.094211 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:40.094237 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 07:14:40.094318 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 07:14:40.094385 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":1,"thread_start_us":145,"threads_started":1}
I20260620 07:14:40.094588 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.090816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:47347 (local address 127.3.245.1:37045)
0620 07:14:40.090955 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:40.090959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:40.090974 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:14:40.091071 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:40.091076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:40.091129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:40.091201 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:40.091460 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.092177 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.093334 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.093981 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.094092 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:40.094177 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:40.094354 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 07:14:40.094433 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:14:40.094705 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.090227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45223 (local address 127.3.245.2:51237)
0620 07:14:40.090396 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:40.090414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:14:40.090502 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:14:40.090890 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:14:40.090894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:14:40.090916 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:14:40.091030 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:40.091036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.092062 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.092066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:40.093340 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:14:40.093351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.094311 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.094316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:14:40.094429 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:40.094450 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:14:40.094521 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 07:14:40.094583 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":23}
I20260620 07:14:40.095225 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.090352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:51237 (local address 127.3.245.3:45223)
0620 07:14:40.090479 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:40.090484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:40.090500 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:14:40.090600 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:40.090603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:40.090655 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:40.090737 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:14:40.091136 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.091964 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.093466 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.094079 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.094171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:40.094232 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:40.095019 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 07:14:40.095116 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:14:40.095336 19586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe804a0922f4ff883878c780de2630d" candidate_uuid: "a845699406eb4de3a8438a6ee4fb6969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c635026273488a943c9e600acafb42" is_pre_election: true
I20260620 07:14:40.095672 19586 raft_consensus.cc:2454] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a845699406eb4de3a8438a6ee4fb6969 in term 0.
I20260620 07:14:40.095747 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe804a0922f4ff883878c780de2630d" candidate_uuid: "a845699406eb4de3a8438a6ee4fb6969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f87febcbf54671bbbbc54c94a7518e" is_pre_election: true
I20260620 07:14:40.096014 19854 raft_consensus.cc:2454] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a845699406eb4de3a8438a6ee4fb6969 in term 0.
I20260620 07:14:40.096139 19656 leader_election.cc:302] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c635026273488a943c9e600acafb42, a845699406eb4de3a8438a6ee4fb6969; no voters:
I20260620 07:14:40.096431 19924 raft_consensus.cc:2781] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:14:40.096517 19924 raft_consensus.cc:482] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:14:40.096567 19924 raft_consensus.cc:3037] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:40.097620 19924 raft_consensus.cc:504] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:40.097997 19924 leader_election.cc:288] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [CANDIDATE]: Term 1 election: Requested vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), 20c635026273488a943c9e600acafb42 (127.3.245.1:37045)
I20260620 07:14:40.098343 19854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe804a0922f4ff883878c780de2630d" candidate_uuid: "a845699406eb4de3a8438a6ee4fb6969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f87febcbf54671bbbbc54c94a7518e"
I20260620 07:14:40.098412 19586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe804a0922f4ff883878c780de2630d" candidate_uuid: "a845699406eb4de3a8438a6ee4fb6969" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c635026273488a943c9e600acafb42"
I20260620 07:14:40.098488 19854 raft_consensus.cc:3037] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:40.098543 19586 raft_consensus.cc:3037] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:14:40.099494 19854 raft_consensus.cc:2454] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a845699406eb4de3a8438a6ee4fb6969 in term 1.
I20260620 07:14:40.099799 19656 leader_election.cc:302] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e; no voters:
I20260620 07:14:40.099825 19586 raft_consensus.cc:2454] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a845699406eb4de3a8438a6ee4fb6969 in term 1.
I20260620 07:14:40.099992 19924 raft_consensus.cc:2781] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:14:40.100225 19924 raft_consensus.cc:686] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 LEADER]: Becoming Leader. State: Replica: a845699406eb4de3a8438a6ee4fb6969, State: Running, Role: LEADER
I20260620 07:14:40.100559 19924 consensus_queue.cc:237] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:14:40.104956 19441 catalog_manager.cc:5495] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 reported cstate change: term changed from 0 to 1, leader changed from <none> to a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2). New cstate: current_term: 1 leader_uuid: "a845699406eb4de3a8438a6ee4fb6969" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } health_report { overall_health: UNKNOWN } } }
W20260620 07:14:40.131201 19901 tablet.cc:2369] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:14:40.134137 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.130555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.3.245.2:44893)
0620 07:14:40.130690 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:14:40.130694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:14:40.130707 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:14:40.130810 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:14:40.130812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:14:40.130867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:14:40.130961 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:14:40.131338 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.131875 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.132785 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:14:40.132993 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:14:40.133073 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:14:40.133348 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:14:40.133753 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:14:40.133869 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:14:40.133917 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 07:14:40.133969 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:14:40.140154 19586 raft_consensus.cc:1261] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Refusing update from remote peer a845699406eb4de3a8438a6ee4fb6969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:40.140156 19854 raft_consensus.cc:1261] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Refusing update from remote peer a845699406eb4de3a8438a6ee4fb6969: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:14:40.141062 19924 consensus_queue.cc:1035] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:40.141346 19927 consensus_queue.cc:1035] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:14:40.150502 19931 mvcc.cc:205] Tried to move back new op lower bound from 7298824929846013952 to 7298824929694416896. Current Snapshot: MvccSnapshot[applied={T|T < 7298824929846013952}]
I20260620 07:14:40.157850 19437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:40.158030 19437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:14:40.160389 19437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Sending DeleteTablet for 3 replicas of tablet bfe804a0922f4ff883878c780de2630d
I20260620 07:14:40.161193 19834 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe804a0922f4ff883878c780de2630d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260620 07:14:40.161307 19700 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe804a0922f4ff883878c780de2630d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:51238
I20260620 07:14:40.161448 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19503
I20260620 07:14:40.162817 19940 tablet_replica.cc:330] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: stopping tablet replica
I20260620 07:14:40.163138 19940 raft_consensus.cc:2229] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:14:40.163484 19940 raft_consensus.cc:2258] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:40.164913 19941 tablet_replica.cc:330] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: stopping tablet replica
I20260620 07:14:40.165203 19940 ts_tablet_manager.cc:1905] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:14:40.165278 19941 raft_consensus.cc:2229] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:14:40.165791 19941 raft_consensus.cc:2258] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:14:40.168025 19940 ts_tablet_manager.cc:1918] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:14:40.168124 19940 log.cc:1196] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/bfe804a0922f4ff883878c780de2630d
I20260620 07:14:40.168457 19940 ts_tablet_manager.cc:1939] T bfe804a0922f4ff883878c780de2630d P b2f87febcbf54671bbbbc54c94a7518e: Deleting consensus metadata
I20260620 07:14:40.169490 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19637
W20260620 07:14:40.169698 19428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37045 (error 108)
I20260620 07:14:40.171200 19428 catalog_manager.cc:4841] TS b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): tablet bfe804a0922f4ff883878c780de2630d (table test-table [id=e04e9b579de944ff9007874a670c3972]) successfully deleted
I20260620 07:14:40.176062 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19771
I20260620 07:14:40.177528 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.177056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44893 (local address 127.0.0.1:51242)
0620 07:14:40.177326 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:40.177414 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:44893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 07:14:40.177824 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:14:40.177119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37045 (local address 127.0.0.1:51914)
0620 07:14:40.177427 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 07:14:40.177502 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260620 07:14:40.177964 19426 catalog_manager.cc:4584] TS a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfe804a0922f4ff883878c780de2630d: Network error: Client connection negotiation failed: client connection to 127.3.245.2:44893: connect: Connection refused (error 111)
I20260620 07:14:40.182301 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19409
I20260620 07:15:00.189267 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--webserver_interface=127.3.245.62
--webserver_port=44509
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:00.309697 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.309983 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.310082 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.314261 19943 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:15:00.314330 19943 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:15:00.314347 19943 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:15:00.314363 19943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:00.314380 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.314399 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.314415 19943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:00.314430 19943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:00.319944 19943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:00.321090 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.322088 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:00.335937 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.335937 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.335944 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.336135 19943 server_base.cc:1034] running on GCE node
I20260620 07:15:00.337072 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:00.338356 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:00.339573 19943 hybrid_clock.cc:648] HybridClock initialized: now 1781939700339533 us; error 99 us; skew 500 ppm
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:00.342875 19943 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:15:00.343984 19943 webserver.cc:460] Webserver started at http://127.3.245.62:44509/ using document root <none> and password file <none>
I20260620 07:15:00.344707 19943 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:00.344827 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:00.349151 19943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:15:00.351518 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:00.353013 19943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:00.353189 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "131b583e25474a1b97eb4d2fd9a3511f"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "91918822b60d24b89cc4dcc7a337a248"
server_key_iv: "f8b9196b549ec7444be0bf1193367897"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.353861 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:00.371682 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:00.375556 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:00.375811 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:00.384421 19943 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45989
I20260620 07:15:00.384781 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45989 every 8 connection(s)
I20260620 07:15:00.385769 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:00.386729 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 19943
I20260620 07:15:00.387171 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:37045
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=34765
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:00.391026 20012 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:15:00.392953 20012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap starting.
I20260620 07:15:00.412365 20012 log.cc:824] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:00.422351 20012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:15:00.422842 20012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap complete.
I20260620 07:15:00.427471 20012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:00.427947 20012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Initialized, Role: FOLLOWER
I20260620 07:15:00.428836 20012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:00.428979 20012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:00.429080 20012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:00.429230 20012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:15:00.431344 20012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:00.431740 20012 leader_election.cc:302] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 131b583e25474a1b97eb4d2fd9a3511f; no voters:
I20260620 07:15:00.432068 20012 leader_election.cc:288] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:15:00.432215 20016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:15:00.432497 20016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 LEADER]: Becoming Leader. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Running, Role: LEADER
I20260620 07:15:00.432934 20016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:00.433415 20012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:00.434448 20017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:15:00.434610 20017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:00.435485 20018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 131b583e25474a1b97eb4d2fd9a3511f. Latest consensus state: current_term: 2 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:15:00.435595 20018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:00.438787 20033 catalog_manager.cc:1259] Loaded cluster ID: c08effe862454fb1979114d578fbc447
I20260620 07:15:00.438881 20033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: loading cluster ID for follower catalog manager: success
I20260620 07:15:00.442826 20033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: acquiring CA information for follower catalog manager: success
I20260620 07:15:00.444720 20033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:15:00.444938 20025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:00.448064 20025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:00.448573 20025 catalog_manager.cc:1259] Loaded cluster ID: c08effe862454fb1979114d578fbc447
I20260620 07:15:00.448700 20025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:00.449492 20025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:00.449982 20025 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Loaded TSK: 0
I20260620 07:15:00.466557 20025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Generated new TSK 1
I20260620 07:15:00.466846 20025 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:15:00.516690 20014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.516978 20014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.517025 20014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.521347 20014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.521421 20014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.521483 20014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:00.526993 20014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:37045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=34765
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:00.528038 20014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.529129 20014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:00.539440 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.539461 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.539688 20014 server_base.cc:1034] running on GCE node
W20260620 07:15:00.539763 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.540177 20014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:00.540956 20014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:00.542217 20014 hybrid_clock.cc:648] HybridClock initialized: now 1781939700542181 us; error 48 us; skew 500 ppm
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:00.545370 20014 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:15:00.546415 20014 webserver.cc:460] Webserver started at http://127.3.245.1:34765/ using document root <none> and password file <none>
I20260620 07:15:00.546995 20014 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:00.547108 20014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:00.551299 20014 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:15:00.553702 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:00.555028 20014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:00.555154 20014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "20c635026273488a943c9e600acafb42"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "0c7db101b0be440ae027dd42eed6cb9e"
server_key_iv: "a5eaf4aab487dceb2c725dc97a15e6ca"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.555732 20014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:00.571370 20014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:00.575054 20014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:00.575285 20014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:00.575960 20014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:00.577291 20057 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:15:00.582618 20014 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:15:00.582789 20014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260620 07:15:00.582912 20014 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:15:00.584887 20014 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:15:00.584962 20014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260620 07:15:00.585263 20057 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Bootstrap starting.
I20260620 07:15:00.595693 20014 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37045
I20260620 07:15:00.595754 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37045 every 8 connection(s)
I20260620 07:15:00.596760 20014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:00.597518 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20014
I20260620 07:15:00.597970 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:44893
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=41245
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:00.607291 20057 log.cc:824] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:00.616117 20057 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:15:00.616592 20057 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Bootstrap complete.
I20260620 07:15:00.616591 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.599038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:53853 (local address 127.3.245.62:45989)
0620 07:15:00.599666 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:00.599688 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:00.600204 (+ 516us) server_negotiation.cc:407] Connection header received
0620 07:15:00.601554 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:00.601570 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:00.602012 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:00.602471 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:00.604081 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.605805 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.607818 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.608512 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.608739 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.611501 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:00.611506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:00.611519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:00.611548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:00.613366 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.613911 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.613916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.613921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.614037 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.614392 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.614396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.614398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.614770 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:00.614913 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:00.615474 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 07:15:00.615758 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":120,"threads_started":1}
I20260620 07:15:00.616603 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.599040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:53853)
0620 07:15:00.599985 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.600047 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:00.601095 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:00.602542 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:00.602551 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:00.603104 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:00.603862 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.603888 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.606001 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.606006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.607652 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.607679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.608695 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.608707 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.608906 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.609569 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:00.609586 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:00.611294 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:00.613496 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.613502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.613514 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.613780 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.614146 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.614149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.614151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.614294 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.614897 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:00.614904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:00.615157 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 07:15:00.615346 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 07:15:00.615684 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":186,"threads_started":1}
I20260620 07:15:00.618203 20057 ts_tablet_manager.cc:1397] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.016s
I20260620 07:15:00.619300 20165 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:15:00.619555 20165 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:00.620041 20165 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:15:00.621378 20057 raft_consensus.cc:348] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:15:00.621939 20057 raft_consensus.cc:729] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c635026273488a943c9e600acafb42, State: Initialized, Role: FOLLOWER
I20260620 07:15:00.622439 20057 consensus_queue.cc:260] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:15:00.622610 19976 ts_manager.cc:194] Registered new tserver with Master: 20c635026273488a943c9e600acafb42 (127.3.245.1:37045)
I20260620 07:15:00.623559 20057 ts_tablet_manager.cc:1428] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 07:15:00.626821 19976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:53853
I20260620 07:15:00.629760 20165 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.1@KRBTEST.COM
I20260620 07:15:00.637300 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.625712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37045 (local address 127.0.0.1:32858)
0620 07:15:00.626214 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.626275 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:00.626415 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:00.627104 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:00.627110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:00.627549 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:00.627693 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.627702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.629055 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.629059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.630047 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.630060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.630971 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.630975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.631093 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.631754 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:00.631766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:00.633203 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:00.635449 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.635457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.635462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.635760 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.636100 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.636103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.636105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.636195 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.636765 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:00.636770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:00.636880 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 07:15:00.637000 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 07:15:00.637161 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":173,"threads_started":1}
I20260620 07:15:00.637439 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.625815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.3.245.1:37045)
0620 07:15:00.626566 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:00.626577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:00.626601 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:15:00.626671 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:00.626677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:00.626907 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:00.627055 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:00.627830 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.628922 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.630222 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.630733 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.630834 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.633345 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:00.633354 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:00.633367 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:00.633396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:00.635328 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.635876 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.635883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.635885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.635978 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.636294 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.636297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.636299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.636592 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:00.636750 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:00.637100 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:15:00.637273 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":116,"threads_started":1}
I20260620 07:15:00.638728 20099 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe804a0922f4ff883878c780de2630d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 07:15:00.639719 20174 tablet_replica.cc:330] stopping tablet replica
I20260620 07:15:00.640028 20174 raft_consensus.cc:2229] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:00.640291 20174 raft_consensus.cc:2258] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:00.641291 20174 ts_tablet_manager.cc:1905] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:00.644852 20174 ts_tablet_manager.cc:1918] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:00.644937 20174 log.cc:1196] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal/wals/bfe804a0922f4ff883878c780de2630d
I20260620 07:15:00.645229 20174 ts_tablet_manager.cc:1939] T bfe804a0922f4ff883878c780de2630d P 20c635026273488a943c9e600acafb42: Deleting consensus metadata
I20260620 07:15:00.646891 19963 catalog_manager.cc:4841] TS 20c635026273488a943c9e600acafb42 (127.3.245.1:37045): tablet bfe804a0922f4ff883878c780de2630d (table test-table [id=e04e9b579de944ff9007874a670c3972]) successfully deleted
W20260620 07:15:00.722025 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.722386 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.722483 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.726734 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.726830 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.726928 20167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:00.732335 20167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:44893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=41245
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:00.733482 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.734630 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:00.744696 20179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.744858 20180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.744809 20167 server_base.cc:1034] running on GCE node
W20260620 07:15:00.744748 20182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.745378 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:00.746057 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:00.747289 20167 hybrid_clock.cc:648] HybridClock initialized: now 1781939700747268 us; error 33 us; skew 500 ppm
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:00.750209 20167 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:15:00.751205 20167 webserver.cc:460] Webserver started at http://127.3.245.2:41245/ using document root <none> and password file <none>
I20260620 07:15:00.751793 20167 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:00.751902 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:00.755829 20167 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:00.757990 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:00.759377 20167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:15:00.759521 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "a845699406eb4de3a8438a6ee4fb6969"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "66754f2e5378ef4af857bbec63926759"
server_key_iv: "129a351fa23ca4376f25f9aef22ba852"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.760084 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:00.802572 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:00.806104 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:00.806322 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:00.806960 20167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:00.808086 20196 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:15:00.812500 20167 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:15:00.812594 20167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 07:15:00.812688 20167 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:15:00.814455 20167 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:15:00.814528 20167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:15:00.814821 20196 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Bootstrap starting.
I20260620 07:15:00.825873 20167 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44893
I20260620 07:15:00.825904 20303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44893 every 8 connection(s)
I20260620 07:15:00.826897 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:00.833213 20196 log.cc:824] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:00.836426 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20167
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:00.836964 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:45223
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=35213
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:00.841286 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.828852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:55011 (local address 127.3.245.62:45989)
0620 07:15:00.829018 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:00.829023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:00.829741 (+ 718us) server_negotiation.cc:407] Connection header received
0620 07:15:00.830662 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:00.830667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:00.830744 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:00.830861 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:00.832104 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.832909 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.834225 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.834557 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.834683 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.837473 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:00.837480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:00.837486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:00.837526 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:00.839108 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.839644 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.839649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.839651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.839737 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.840106 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.840109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.840111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.840300 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:00.840407 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:00.840968 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 07:15:00.841110 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:15:00.841475 20196 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:15:00.841884 20196 tablet_bootstrap.cc:492] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Bootstrap complete.
I20260620 07:15:00.842149 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.829029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:55011)
0620 07:15:00.829534 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.829587 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:00.830452 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:00.830997 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:00.831003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:00.831377 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:00.831946 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.831961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.833104 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.833107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.834031 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.834042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.834690 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.834693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.834885 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.835494 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:00.835510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:00.837271 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:00.839238 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.839244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.839257 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.839532 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.839850 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.839854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.839856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.839998 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.840406 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:00.840412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:00.840724 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 07:15:00.840925 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 07:15:00.841243 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260620 07:15:00.843230 20196 ts_tablet_manager.cc:1397] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.013s
I20260620 07:15:00.843418 20304 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:15:00.843641 20304 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:00.844113 20304 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:15:00.845774 19976 ts_manager.cc:194] Registered new tserver with Master: a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
I20260620 07:15:00.847015 19976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:55011
I20260620 07:15:00.847693 20196 raft_consensus.cc:348] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:15:00.848428 20196 raft_consensus.cc:729] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845699406eb4de3a8438a6ee4fb6969, State: Initialized, Role: FOLLOWER
I20260620 07:15:00.849074 20196 consensus_queue.cc:260] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } }
I20260620 07:15:00.849830 20304 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
I20260620 07:15:00.850327 20196 ts_tablet_manager.cc:1428] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.2@KRBTEST.COM
I20260620 07:15:00.857978 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.846636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44893 (local address 127.0.0.1:53876)
0620 07:15:00.846782 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:00.846802 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:00.846896 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:00.847916 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:00.847919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:00.848106 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:00.848206 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.848213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.849663 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.849666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.850507 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:00.850515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.851895 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.851899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:00.851990 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.852558 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:00.852579 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:00.854151 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:00.856067 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.856073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.856075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.856293 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.856674 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:00.856677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:00.856679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:00.856771 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:00.857343 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:00.857346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:00.857464 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 07:15:00.857616 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 07:15:00.857784 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 07:15:00.858115 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:00.846902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.3.245.2:44893)
0620 07:15:00.847469 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:00.847480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:00.847501 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:15:00.847559 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:00.847564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:00.847760 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:00.848363 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:00.848499 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.849531 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.851300 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:00.851790 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:00.851886 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:00.854304 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:00.854315 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:00.854329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:00.854362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:00.855940 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.856416 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.856422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.856424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.856517 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:00.856879 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:00.856883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:00.856885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:00.857195 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:00.857390 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:00.857709 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 07:15:00.857914 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":105,"threads_started":1}
I20260620 07:15:00.859252 20238 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe804a0922f4ff883878c780de2630d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:14:40 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:53876
I20260620 07:15:00.860240 20311 tablet_replica.cc:330] stopping tablet replica
I20260620 07:15:00.860598 20311 raft_consensus.cc:2229] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:00.860908 20311 raft_consensus.cc:2258] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:00.861865 20311 ts_tablet_manager.cc:1905] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:00.865629 20311 ts_tablet_manager.cc:1918] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:00.865756 20311 log.cc:1196] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/bfe804a0922f4ff883878c780de2630d
I20260620 07:15:00.866020 20311 ts_tablet_manager.cc:1939] T bfe804a0922f4ff883878c780de2630d P a845699406eb4de3a8438a6ee4fb6969: Deleting consensus metadata
I20260620 07:15:00.867565 19961 catalog_manager.cc:4841] TS a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): tablet bfe804a0922f4ff883878c780de2630d (table test-table [id=e04e9b579de944ff9007874a670c3972]) successfully deleted
W20260620 07:15:00.958977 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:00.959295 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:00.959362 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:00.963845 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:00.963953 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:00.964041 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:00.970218 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:45223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=35213
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:00.971349 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:00.972404 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:00.982493 20317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:00.982532 20316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.982668 20308 server_base.cc:1034] running on GCE node
W20260620 07:15:00.982494 20319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:00.983271 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:00.983997 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:00.985248 20308 hybrid_clock.cc:648] HybridClock initialized: now 1781939700985218 us; error 38 us; skew 500 ppm
Jun 20 07:15:00 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:00.988045 20308 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:15:00.989043 20308 webserver.cc:460] Webserver started at http://127.3.245.3:35213/ using document root <none> and password file <none>
I20260620 07:15:00.989568 20308 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:00.989640 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:00.993536 20308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:00.995728 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:00.997282 20308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:00.997429 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b2f87febcbf54671bbbbc54c94a7518e"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "8ac6575d47666954bc5e9ea5ddbb75c3"
server_key_iv: "0bf1ae32bea4c9dead955132a314a3ce"
server_key_version: "encryptionkey@0"
I20260620 07:15:00.998078 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:01.009855 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:01.013381 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:01.013613 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:01.014279 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:01.015187 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:01.015259 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:01.015336 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:01.015378 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:01.025108 20308 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:45223
I20260620 07:15:01.025138 20439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:45223 every 8 connection(s)
I20260620 07:15:01.026232 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:01.026937 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20308
I20260620 07:15:01.036728 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.027902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.3.245.62:45989)
0620 07:15:01.028070 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.028075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.028289 (+ 214us) server_negotiation.cc:407] Connection header received
0620 07:15:01.028396 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.028398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.028447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.028577 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.029483 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.030011 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.030886 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.031113 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.031219 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.032695 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.032699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.032702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.032729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.034592 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.035246 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.035250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.035251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.035330 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.035627 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.035630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.035632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.035890 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.036094 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.036375 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.036508 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":56}
Jun 20 07:15:01 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:01.041990 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.028704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:37027 (local address 127.3.245.62:45989)
0620 07:15:01.028953 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.028959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.030163 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 07:15:01.031060 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.031063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.031112 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.031207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.032425 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.033095 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.034947 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.035343 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.035463 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.038280 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.038284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.038287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.038311 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.039799 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.040333 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.040337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.040339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.040421 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.040846 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.040850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.040852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.041040 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.041147 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.041674 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.041814 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260620 07:15:01.042829 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.029193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:37027)
0620 07:15:01.029962 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.030014 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.030856 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.031389 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.031396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.031757 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:01.032275 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.032289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.033241 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.033261 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.034792 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.034804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.035513 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.035516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.035720 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.036393 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:01.036413 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:01.038104 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:01.039926 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.039932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.039950 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.040218 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.040516 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:01.040519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:01.040521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:01.040748 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:01.041146 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:01.041154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:01.041427 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.041618 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.041987 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":155,"threads_started":1}
I20260620 07:15:01.044312 20440 heartbeater.cc:347] Connected to a master server at 127.3.245.62:45989
I20260620 07:15:01.044591 20440 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:01.045181 20440 heartbeater.cc:511] Master 127.3.245.62:45989 requested a full tablet report, sending...
I20260620 07:15:01.046247 19976 ts_manager.cc:194] Registered new tserver with Master: b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223)
I20260620 07:15:01.046814 19976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:37027
I20260620 07:15:01.051337 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:01.055936 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.052096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.3.245.62:45989)
0620 07:15:01.052273 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.052278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.052510 (+ 232us) server_negotiation.cc:407] Connection header received
0620 07:15:01.052683 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.052691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.052812 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.052903 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:01.053232 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.053786 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.054687 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.054921 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.055028 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.055352 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:01.055495 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 07:15:01.055783 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55596: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 07:15:01.056035 20443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55596: token expired
W20260620 07:15:01.056385 19906 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.245.62:45989)
I20260620 07:15:01.064954 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.056878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.3.245.62:45989)
0620 07:15:01.057000 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.057003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.057016 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:01.057124 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.057126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.057168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.057253 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:01.057979 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.058522 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.059371 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.059589 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.059739 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.061307 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:01.061310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:01.061312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:01.061335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:01.062881 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.063446 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.063449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.063451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.063535 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:01.063940 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:01.063943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:01.063945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:01.064162 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:01.064330 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.064582 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.064792 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:15:01.067761 19976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:01.071254 19976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:01.079939 20238 tablet_service.cc:1467] Processing CreateTablet for tablet 188c4f0975c348679f7980863899551b (DEFAULT_TABLE table=test-table [id=b8083c73cd9c433cbabf3548ea8442d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:01.079939 20099 tablet_service.cc:1467] Processing CreateTablet for tablet 188c4f0975c348679f7980863899551b (DEFAULT_TABLE table=test-table [id=b8083c73cd9c433cbabf3548ea8442d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:01.080652 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188c4f0975c348679f7980863899551b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:01.080659 20099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188c4f0975c348679f7980863899551b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:01.083628 20196 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Bootstrap starting.
I20260620 07:15:01.084079 20057 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Bootstrap starting.
I20260620 07:15:01.084549 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.079077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45223 (local address 127.0.0.1:32922)
0620 07:15:01.079200 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.079220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.079302 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.080282 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.080285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.080304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:01.080420 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.080427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.081871 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.081878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.082969 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.082980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.084071 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.084074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.084174 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.084229 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.084309 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.084399 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 07:15:01.084648 20196 tablet_bootstrap.cc:654] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.084990 20057 tablet_bootstrap.cc:654] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.085028 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.079257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.3.245.3:45223)
0620 07:15:01.079747 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.079756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.079822 (+ 66us) server_negotiation.cc:407] Connection header received
0620 07:15:01.079891 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.079896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.080080 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.080591 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:01.080834 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.081740 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.083108 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.083876 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.083991 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.084147 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.084348 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.084454 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":177,"threads_started":1}
I20260620 07:15:01.086483 20196 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: No bootstrap required, opened a new log
I20260620 07:15:01.086500 20057 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: No bootstrap required, opened a new log
I20260620 07:15:01.086584 20196 ts_tablet_manager.cc:1397] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:01.086608 20057 ts_tablet_manager.cc:1397] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:01.087046 20196 raft_consensus.cc:348] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.087054 20057 raft_consensus.cc:348] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.087143 20196 raft_consensus.cc:374] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:01.087172 20196 raft_consensus.cc:729] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a845699406eb4de3a8438a6ee4fb6969, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.087167 20057 raft_consensus.cc:374] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:01.087214 20057 raft_consensus.cc:729] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c635026273488a943c9e600acafb42, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.087324 20196 consensus_queue.cc:260] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.087333 20057 consensus_queue.cc:260] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.087584 20057 ts_tablet_manager.cc:1428] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:15:01.087993 20196 ts_tablet_manager.cc:1428] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 07:15:01.089594 20374 tablet_service.cc:1467] Processing CreateTablet for tablet 188c4f0975c348679f7980863899551b (DEFAULT_TABLE table=test-table [id=b8083c73cd9c433cbabf3548ea8442d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:01.090873 20374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188c4f0975c348679f7980863899551b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 07:15:01.097925 20166 tablet.cc:2369] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:01.098407 20452 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Bootstrap starting.
I20260620 07:15:01.100979 20452 tablet_bootstrap.cc:654] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:01.102299 20452 log.cc:824] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:01.104707 20452 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: No bootstrap required, opened a new log
I20260620 07:15:01.105010 20452 ts_tablet_manager.cc:1397] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:15:01.108537 20452 raft_consensus.cc:348] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.108927 20452 raft_consensus.cc:374] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:01.109025 20452 raft_consensus.cc:729] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f87febcbf54671bbbbc54c94a7518e, State: Initialized, Role: FOLLOWER
I20260620 07:15:01.109572 20452 consensus_queue.cc:260] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.110431 20440 heartbeater.cc:503] Master 127.3.245.62:45989 was elected leader, sending a full tablet report...
I20260620 07:15:01.110719 20452 ts_tablet_manager.cc:1428] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 07:15:01.277439 20441 tablet.cc:2369] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:01.277861 20171 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:01.278128 20171 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.279541 20171 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
I20260620 07:15:01.283866 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.279919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:52357 (local address 127.3.245.2:44893)
0620 07:15:01.280060 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.280065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.280361 (+ 296us) server_negotiation.cc:407] Connection header received
0620 07:15:01.280530 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.280534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.280599 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.280732 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:01.281060 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.281681 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.282821 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.283368 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.283469 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.283652 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.283689 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.283761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:15:01.283877 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.279936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:34987 (local address 127.3.245.3:45223)
0620 07:15:01.280067 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.280074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.280224 (+ 150us) server_negotiation.cc:407] Connection header received
0620 07:15:01.280386 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.280389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.280449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.280529 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:01.280906 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.281460 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.282740 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.283181 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.283267 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.283324 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.283615 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.283703 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:15:01.283855 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.279779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45223 (local address 127.3.245.1:34987)
0620 07:15:01.280129 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.280142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.280260 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.280551 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.280554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.280570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:01.280745 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.280753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.281607 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.281611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.282605 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.282616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.283317 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.283321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.283451 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.283511 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.283584 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.283670 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 07:15:01.283886 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.279787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44893 (local address 127.3.245.1:52357)
0620 07:15:01.280249 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.280263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:01.280395 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:01.280775 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:01.280778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:01.280794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:01.280938 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.280949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.281859 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.281866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.282658 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:01.282693 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.283484 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.283487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:01.283550 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.283595 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 07:15:01.283657 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 07:15:01.283721 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":68,"thread_start_us":79,"threads_started":1}
I20260620 07:15:01.284528 20394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188c4f0975c348679f7980863899551b" candidate_uuid: "20c635026273488a943c9e600acafb42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f87febcbf54671bbbbc54c94a7518e" is_pre_election: true
I20260620 07:15:01.284566 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188c4f0975c348679f7980863899551b" candidate_uuid: "20c635026273488a943c9e600acafb42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845699406eb4de3a8438a6ee4fb6969" is_pre_election: true
I20260620 07:15:01.284905 20394 raft_consensus.cc:2454] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c635026273488a943c9e600acafb42 in term 0.
I20260620 07:15:01.284909 20258 raft_consensus.cc:2454] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c635026273488a943c9e600acafb42 in term 0.
I20260620 07:15:01.285449 20054 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20c635026273488a943c9e600acafb42, a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e; no voters:
I20260620 07:15:01.285744 20171 raft_consensus.cc:2781] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:01.285835 20171 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:01.285889 20171 raft_consensus.cc:3037] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.286943 20171 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.287369 20171 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 1 election: Requested vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
I20260620 07:15:01.287766 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188c4f0975c348679f7980863899551b" candidate_uuid: "20c635026273488a943c9e600acafb42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a845699406eb4de3a8438a6ee4fb6969"
I20260620 07:15:01.287943 20258 raft_consensus.cc:3037] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.287998 20394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188c4f0975c348679f7980863899551b" candidate_uuid: "20c635026273488a943c9e600acafb42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f87febcbf54671bbbbc54c94a7518e"
I20260620 07:15:01.288143 20394 raft_consensus.cc:3037] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:01.289251 20258 raft_consensus.cc:2454] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c635026273488a943c9e600acafb42 in term 1.
I20260620 07:15:01.289314 20394 raft_consensus.cc:2454] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c635026273488a943c9e600acafb42 in term 1.
I20260620 07:15:01.289616 20052 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c635026273488a943c9e600acafb42, a845699406eb4de3a8438a6ee4fb6969; no voters:
I20260620 07:15:01.289906 20171 raft_consensus.cc:2781] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:01.290099 20171 raft_consensus.cc:686] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 LEADER]: Becoming Leader. State: Replica: 20c635026273488a943c9e600acafb42, State: Running, Role: LEADER
I20260620 07:15:01.290365 20171 consensus_queue.cc:237] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:01.293685 19976 catalog_manager.cc:5495] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20c635026273488a943c9e600acafb42 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "20c635026273488a943c9e600acafb42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } health_report { overall_health: UNKNOWN } } }
W20260620 07:15:01.328320 20305 tablet.cc:2369] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:01.336376 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.332009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.3.245.1:37045)
0620 07:15:01.332343 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:01.332350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:01.332363 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:01.332441 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:01.332443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:01.332521 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:01.332709 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:01.333012 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.333639 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.334625 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:01.334882 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:01.334967 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:01.335260 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:01.335876 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:01.335970 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:01.336097 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 07:15:01.336152 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":175,"threads_started":1}
I20260620 07:15:01.340719 20394 raft_consensus.cc:1261] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Refusing update from remote peer 20c635026273488a943c9e600acafb42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:01.340723 20258 raft_consensus.cc:1261] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Refusing update from remote peer 20c635026273488a943c9e600acafb42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:01.341460 20171 consensus_queue.cc:1035] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:01.341689 20457 consensus_queue.cc:1035] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:01.348202 20461 mvcc.cc:205] Tried to move back new op lower bound from 7298825016684683264 to 7298825016487772160. Current Snapshot: MvccSnapshot[applied={T|T < 7298825016684683264}]
I20260620 07:15:01.348909 20463 mvcc.cc:205] Tried to move back new op lower bound from 7298825016684683264 to 7298825016487772160. Current Snapshot: MvccSnapshot[applied={T|T < 7298825016684683264}]
I20260620 07:15:01.356868 19975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:01.357069 19975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:01.359758 19975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Sending DeleteTablet for 3 replicas of tablet 188c4f0975c348679f7980863899551b
I20260620 07:15:01.360567 20374 tablet_service.cc:1514] Processing DeleteTablet for tablet 188c4f0975c348679f7980863899551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:01 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:32922
I20260620 07:15:01.360716 20238 tablet_service.cc:1514] Processing DeleteTablet for tablet 188c4f0975c348679f7980863899551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:01 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:53876
I20260620 07:15:01.360945 20311 tablet_replica.cc:330] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: stopping tablet replica
I20260620 07:15:01.361085 20311 raft_consensus.cc:2229] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:01.361112 20099 tablet_service.cc:1514] Processing DeleteTablet for tablet 188c4f0975c348679f7980863899551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:01 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 07:15:01.361195 20470 tablet_replica.cc:330] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: stopping tablet replica
I20260620 07:15:01.361249 20311 raft_consensus.cc:2258] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:01.361534 20470 raft_consensus.cc:2229] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:01.361675 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20014
I20260620 07:15:01.362362 20311 ts_tablet_manager.cc:1905] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:01.362634 20470 raft_consensus.cc:2258] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:01.364148 20470 ts_tablet_manager.cc:1905] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:01.365798 20311 ts_tablet_manager.cc:1918] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:01.365893 20311 log.cc:1196] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/188c4f0975c348679f7980863899551b
I20260620 07:15:01.366228 20311 ts_tablet_manager.cc:1939] T 188c4f0975c348679f7980863899551b P a845699406eb4de3a8438a6ee4fb6969: Deleting consensus metadata
I20260620 07:15:01.366710 20470 ts_tablet_manager.cc:1918] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:01.366829 20470 log.cc:1196] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/188c4f0975c348679f7980863899551b
I20260620 07:15:01.367081 19961 catalog_manager.cc:4841] TS a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): tablet 188c4f0975c348679f7980863899551b (table test-table [id=b8083c73cd9c433cbabf3548ea8442d7]) successfully deleted
I20260620 07:15:01.367223 20470 ts_tablet_manager.cc:1939] T 188c4f0975c348679f7980863899551b P b2f87febcbf54671bbbbc54c94a7518e: Deleting consensus metadata
I20260620 07:15:01.368374 19963 catalog_manager.cc:4841] TS b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): tablet 188c4f0975c348679f7980863899551b (table test-table [id=b8083c73cd9c433cbabf3548ea8442d7]) successfully deleted
W20260620 07:15:01.370528 19963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37045 (error 108)
I20260620 07:15:01.370595 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20167
I20260620 07:15:01.372100 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:01.371802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37045 (local address 127.0.0.1:32878)
0620 07:15:01.371954 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:01.372029 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 07:15:01.372287 19963 catalog_manager.cc:4584] TS 20c635026273488a943c9e600acafb42 (127.3.245.1:37045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 188c4f0975c348679f7980863899551b: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37045: connect: Connection refused (error 111)
I20260620 07:15:01.377454 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20308
I20260620 07:15:01.383626 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 19943
I20260620 07:15:21.390873 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--webserver_interface=127.3.245.62
--webserver_port=44509
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:21.505435 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:21.505738 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:21.505779 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:21.509997 20473 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:15:21.510068 20473 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:15:21.510115 20473 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:15:21.510138 20473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:21.510177 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:21.510205 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:21.510243 20473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:21.510335 20473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:21.516007 20473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=44509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:21.517071 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:21.518052 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:15:21.527123 20473 server_base.cc:1034] running on GCE node
W20260620 07:15:21.527060 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:21.527056 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:21.527081 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:21.527971 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:21.529191 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:21.530396 20473 hybrid_clock.cc:648] HybridClock initialized: now 1781939721530377 us; error 32 us; skew 500 ppm
Jun 20 07:15:21 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:21.533370 20473 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:15:21.534348 20473 webserver.cc:460] Webserver started at http://127.3.245.62:44509/ using document root <none> and password file <none>
I20260620 07:15:21.535046 20473 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:21.535138 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:21.539039 20473 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:15:21.541237 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:21.542557 20473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:21.542716 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "131b583e25474a1b97eb4d2fd9a3511f"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "91918822b60d24b89cc4dcc7a337a248"
server_key_iv: "f8b9196b549ec7444be0bf1193367897"
server_key_version: "encryptionkey@0"
I20260620 07:15:21.543289 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:21.573237 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:21.576654 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:21.576895 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:21.584512 20473 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:45989
I20260620 07:15:21.584843 20540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:45989 every 8 connection(s)
I20260620 07:15:21.585865 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:21.588093 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20473
I20260620 07:15:21.588598 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:37045
--local_ip_for_outbound_sockets=127.3.245.1
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=34765
--webserver_interface=127.3.245.1
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:21.590958 20541 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:15:21.592849 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap starting.
I20260620 07:15:21.610302 20541 log.cc:824] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:21.621977 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:15:21.622381 20541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Bootstrap complete.
I20260620 07:15:21.626516 20541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:21.626894 20541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Initialized, Role: FOLLOWER
I20260620 07:15:21.627363 20541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:21.627476 20541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:21.627564 20541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:21.627679 20541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:15:21.629611 20541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:21.629953 20541 leader_election.cc:302] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 131b583e25474a1b97eb4d2fd9a3511f; no voters:
I20260620 07:15:21.630302 20541 leader_election.cc:288] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:15:21.630375 20545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:15:21.630623 20545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [term 3 LEADER]: Becoming Leader. State: Replica: 131b583e25474a1b97eb4d2fd9a3511f, State: Running, Role: LEADER
I20260620 07:15:21.630965 20545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } }
I20260620 07:15:21.631539 20541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:21.632472 20546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:15:21.632656 20546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:21.632978 20547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 131b583e25474a1b97eb4d2fd9a3511f. Latest consensus state: current_term: 3 leader_uuid: "131b583e25474a1b97eb4d2fd9a3511f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131b583e25474a1b97eb4d2fd9a3511f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 45989 } } }
I20260620 07:15:21.633121 20547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:21.633433 20554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:21.638633 20554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:21.639254 20554 catalog_manager.cc:1259] Loaded cluster ID: c08effe862454fb1979114d578fbc447
I20260620 07:15:21.639317 20554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:21.641763 20554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:21.642457 20554 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Loaded TSK: 0
I20260620 07:15:21.642539 20554 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 131b583e25474a1b97eb4d2fd9a3511f: Loaded TSK: 1
F20260620 07:15:21.643316 20554 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe140dc39ad google::LogMessage::Fail() at ??:0
@ 0x7fe140dc688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe140dc34de google::LogMessage::Flush() at ??:0
@ 0x7fe140dc4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe143a8718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe143a7ff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe14b7287bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe14b70299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe14b703178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fe14b744df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe14b76382b std::function<>::operator()() at ??:0
@ 0x7fe14b702b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe14b703f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe14b700579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe14b744640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe1430153d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe143015c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe1430173f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe143006512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe14464a6db start_thread at ??:0
@ 0x7fe13f8c161f clone at ??:0
W20260620 07:15:21.734658 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:21.735061 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:21.735177 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:21.742044 20543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:21.742138 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:21.742228 20543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:21.750474 20543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:37045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=34765
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:21.752096 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:21.753583 20543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:21.766342 20588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:21.766475 20589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:21.766312 20591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:21.766857 20543 server_base.cc:1034] running on GCE node
I20260620 07:15:21.767290 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:21.768003 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:21.769214 20543 hybrid_clock.cc:648] HybridClock initialized: now 1781939721769200 us; error 28 us; skew 500 ppm
Jun 20 07:15:21 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:21.772331 20543 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:15:21.773435 20543 webserver.cc:460] Webserver started at http://127.3.245.1:34765/ using document root <none> and password file <none>
I20260620 07:15:21.774067 20543 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:21.774134 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:21.778628 20543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 07:15:21.781287 20598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:21.782845 20543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:21.783015 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "20c635026273488a943c9e600acafb42"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "0c7db101b0be440ae027dd42eed6cb9e"
server_key_iv: "a5eaf4aab487dceb2c725dc97a15e6ca"
server_key_version: "encryptionkey@0"
I20260620 07:15:21.783628 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:21.818286 20543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:21.821801 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:21.822027 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:21.822666 20543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:21.823843 20605 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 07:15:21.829416 20543 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:15:21.829520 20543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 07:15:21.829631 20543 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:15:21.831496 20543 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:15:21.831568 20543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 07:15:21.831845 20605 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Bootstrap starting.
I20260620 07:15:21.843945 20543 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37045
I20260620 07:15:21.845412 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:21.846740 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20543
I20260620 07:15:21.846341 20712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37045 every 8 connection(s)
I20260620 07:15:21.847267 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:44893
--local_ip_for_outbound_sockets=127.3.245.2
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=41245
--webserver_interface=127.3.245.2
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:21.858156 20605 log.cc:824] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:21.868496 20605 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:15:21.869035 20605 tablet_bootstrap.cc:492] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Bootstrap complete.
I20260620 07:15:21.870652 20605 ts_tablet_manager.cc:1397] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.007s
I20260620 07:15:21.874276 20605 raft_consensus.cc:348] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:21.874877 20605 raft_consensus.cc:729] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c635026273488a943c9e600acafb42, State: Initialized, Role: FOLLOWER
I20260620 07:15:21.875465 20605 consensus_queue.cc:260] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:21.876686 20605 ts_tablet_manager.cc:1428] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260620 07:15:21.886929 20715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:15:21.847961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:57755)
0620 07:15:21.849263 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:21.849331 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:21.850597 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:21.886072 (+ 35475us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":137,"threads_started":1}
W20260620 07:15:21.887446 20713 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45989 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45989: Client connection negotiation failed: client connection to 127.3.245.62:45989: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:15:21.888307 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:21.888031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:44263)
0620 07:15:21.888179 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:21.888247 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:15:21.889194 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:21.889000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:35455)
0620 07:15:21.889103 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:21.889145 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:15:21.889406 20713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:15:21.979333 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:21.979622 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:21.979676 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:21.983822 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:21.983901 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:21.983999 20716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:21.989720 20716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:44893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=41245
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:21.990847 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:21.991889 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:22.001618 20723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:22.001610 20726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:22.001760 20724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:22.002071 20716 server_base.cc:1034] running on GCE node
I20260620 07:15:22.002556 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:22.003356 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:22.004555 20716 hybrid_clock.cc:648] HybridClock initialized: now 1781939722004517 us; error 54 us; skew 500 ppm
Jun 20 07:15:22 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:22.007529 20716 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:15:22.008411 20716 webserver.cc:460] Webserver started at http://127.3.245.2:41245/ using document root <none> and password file <none>
I20260620 07:15:22.009016 20716 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:22.009079 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:22.013228 20716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:15:22.015492 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.016995 20716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:22.017104 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "a845699406eb4de3a8438a6ee4fb6969"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "66754f2e5378ef4af857bbec63926759"
server_key_iv: "129a351fa23ca4376f25f9aef22ba852"
server_key_version: "encryptionkey@0"
I20260620 07:15:22.017699 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:22.036020 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:22.039464 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:22.039692 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:22.040319 20716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:22.041188 20716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:22.041234 20716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.041303 20716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:22.041338 20716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.050653 20716 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:44893
I20260620 07:15:22.050668 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:44893 every 8 connection(s)
I20260620 07:15:22.051738 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:22.055473 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.053956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:52279)
0620 07:15:22.054649 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.054759 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":113,"threads_started":1}
W20260620 07:15:22.055970 20847 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45989 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45989: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
I20260620 07:15:22.056674 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.056378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:42095)
0620 07:15:22.056503 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.056571 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:15:22.056787 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20716
I20260620 07:15:22.057271 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:45223
--local_ip_for_outbound_sockets=127.3.245.3
--tserver_master_addrs=127.3.245.62:45989
--webserver_port=35213
--webserver_interface=127.3.245.3
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:35847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:15:22.057356 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.057114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:37393)
0620 07:15:22.057239 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.057296 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 07:15:22.057585 20847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:15:22.168918 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:22.169185 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:22.169278 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:22.173527 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:22.173591 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:22.173652 20850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:22.178918 20850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:45223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=35213
--tserver_master_addrs=127.3.245.62:45989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:22.179942 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:22.181017 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:15:22.190742 20850 server_base.cc:1034] running on GCE node
W20260620 07:15:22.190688 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:22.190689 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:22.190688 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:22.191375 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:22.192024 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:22.193213 20850 hybrid_clock.cc:648] HybridClock initialized: now 1781939722193178 us; error 52 us; skew 500 ppm
Jun 20 07:15:22 dist-test-slave-ldfr krb5kdc[19396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939722, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:22.195936 20850 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:15:22.196902 20850 webserver.cc:460] Webserver started at http://127.3.245.3:35213/ using document root <none> and password file <none>
I20260620 07:15:22.197499 20850 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:22.197592 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:22.201445 20850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 07:15:22.203490 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.204712 20850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:22.204840 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "b2f87febcbf54671bbbbc54c94a7518e"
format_stamp: "Formatted at 2026-06-20 07:14:39 on dist-test-slave-ldfr"
server_key: "8ac6575d47666954bc5e9ea5ddbb75c3"
server_key_iv: "0bf1ae32bea4c9dead955132a314a3ce"
server_key_version: "encryptionkey@0"
I20260620 07:15:22.205396 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:22.217880 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:22.220865 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:22.221072 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:22.221674 20850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:22.222565 20850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:22.222635 20850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.222700 20850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:22.222751 20850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:22.232144 20850 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:45223
I20260620 07:15:22.232167 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:45223 every 8 connection(s)
I20260620 07:15:22.233201 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:22.233857 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 20850
I20260620 07:15:22.237051 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.235423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:35777)
0620 07:15:22.236068 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.236168 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":108,"threads_started":1}
W20260620 07:15:22.237510 20979 heartbeater.cc:644] Failed to heartbeat to 127.3.245.62:45989 (0 consecutive failures): Network error: Failed to ping master at 127.3.245.62:45989: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
I20260620 07:15:22.238088 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.237889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:51157)
0620 07:15:22.237989 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.238035 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:15:22.238703 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.238533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:32893)
0620 07:15:22.238619 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.238659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:15:22.238921 20979 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:15:22.890358 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:22.889948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:47053)
0620 07:15:22.890189 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:22.890246 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260620 07:15:23.058686 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.058207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:50067)
0620 07:15:23.058512 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:23.058576 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260620 07:15:23.079598 20985 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:23.079852 20985 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:23.081074 20985 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
Jun 20 07:15:23 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.3@KRBTEST.COM
Jun 20 07:15:23 dist-test-slave-ldfr krb5kdc[19396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939721, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.2@KRBTEST.COM
I20260620 07:15:23.098587 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.081294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45223 (local address 127.3.245.1:56597)
0620 07:15:23.081518 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:23.081532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:23.081652 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:23.083718 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:23.083723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:23.084099 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:23.084588 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:23.084603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.086311 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.086314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:23.087839 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:23.087851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.088981 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.088985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:23.089352 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:23.090238 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:23.090283 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:23.092350 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:23.095182 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:23.095189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:23.095207 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:23.095590 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:23.096107 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:23.096110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:23.096112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:23.096307 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:23.097503 (+ 1196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:23.097511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:23.097833 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 07:15:23.098040 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 07:15:23.098395 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":295,"spinlock_wait_cycles":46464}
I20260620 07:15:23.098892 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.081494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:39035 (local address 127.3.245.2:44893)
0620 07:15:23.081928 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:23.081951 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:23.082141 (+ 190us) server_negotiation.cc:407] Connection header received
0620 07:15:23.082918 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:23.082926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:23.083287 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:23.083649 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:23.084831 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.086076 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.088000 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.088757 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.088942 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:23.092946 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:23.092951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:23.092964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:23.093016 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:23.095392 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:23.095977 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:23.095986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:23.095991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:23.096200 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:23.096743 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:23.096747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:23.096748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:23.097374 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:23.097638 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:23.098465 (+ 827us) server_negotiation.cc:299] Negotiation successful
0620 07:15:23.098708 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":82,"threads_started":1}
I20260620 07:15:23.098925 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.081445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:56597 (local address 127.3.245.3:45223)
0620 07:15:23.081825 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:23.081842 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:23.081887 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:15:23.082801 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:23.082810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:23.083287 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:23.083649 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:23.084888 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.086164 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.087975 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.088757 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.088942 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:23.092685 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:23.092693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:23.092709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:23.092742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:23.095026 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:23.095732 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:23.095739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:23.095792 (+ 53us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:23.095981 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:23.096740 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:23.096745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:23.096746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:23.097363 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:23.097517 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:23.098257 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 07:15:23.098691 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260620 07:15:23.099779 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.081314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:44893 (local address 127.3.245.1:39035)
0620 07:15:23.082031 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:23.082043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:23.082136 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:23.084148 (+ 2012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:23.084150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:23.084326 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:23.084588 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:23.084603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.086229 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.086233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:23.087839 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:23.087851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:23.088981 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:23.088985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:23.089350 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:23.090238 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:23.090266 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:23.092693 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:23.095527 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:23.095530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:23.095533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:23.095810 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:23.096347 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:23.096353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:23.096355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:23.096527 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:23.097523 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:23.097527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:23.097834 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 07:15:23.098041 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 07:15:23.099618 (+ 1577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":2010,"spinlock_wait_cycles":13440,"thread_start_us":54,"threads_started":1}
W20260620 07:15:23.102520 20602 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
W20260620 07:15:23.102715 20600 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
I20260620 07:15:23.102787 20602 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c635026273488a943c9e600acafb42; no voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e
I20260620 07:15:23.103298 20985 raft_consensus.cc:2726] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:15:23.239883 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.239463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:57419)
0620 07:15:23.239726 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:23.239787 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260620 07:15:23.891610 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:23.891166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:51707)
0620 07:15:23.891446 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:23.891515 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260620 07:15:24.059815 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:24.059414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:45313)
0620 07:15:24.059664 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:24.059722 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260620 07:15:24.241276 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:24.240674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:44553)
0620 07:15:24.241036 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:24.241116 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":75,"threads_started":1}
I20260620 07:15:24.672094 20993 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:24.672232 20993 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:24.672858 20993 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
W20260620 07:15:24.673715 20602 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
W20260620 07:15:24.673804 20600 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
I20260620 07:15:24.673933 20600 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c635026273488a943c9e600acafb42; no voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e
I20260620 07:15:24.674161 20993 raft_consensus.cc:2726] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:15:24.893080 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:24.892469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:49305)
0620 07:15:24.892910 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:24.892979 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":76,"threads_started":1}
I20260620 07:15:25.061177 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:25.060699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:43749)
0620 07:15:25.061002 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:25.061063 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260620 07:15:25.242504 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:25.242072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:33243)
0620 07:15:25.242350 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:25.242407 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":106,"threads_started":1}
I20260620 07:15:25.894419 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:25.893918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:51517)
0620 07:15:25.894245 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:25.894317 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
I20260620 07:15:26.062573 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:26.062106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:59363)
0620 07:15:26.062417 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:26.062479 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260620 07:15:26.243829 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:26.243286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:35433)
0620 07:15:26.243592 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:26.243702 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260620 07:15:26.896021 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:26.895477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:33129)
0620 07:15:26.895864 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:26.895925 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260620 07:15:27.024008 21001 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:27.024139 21001 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:27.024588 21001 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
W20260620 07:15:27.025403 20602 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
W20260620 07:15:27.025494 20600 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
I20260620 07:15:27.025561 20600 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c635026273488a943c9e600acafb42; no voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e
I20260620 07:15:27.025692 21001 raft_consensus.cc:2726] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:15:27.064051 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:27.063529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:37931)
0620 07:15:27.063884 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:27.063950 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":95,"threads_started":1}
I20260620 07:15:27.245119 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:27.244687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:43483)
0620 07:15:27.244969 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:27.245024 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260620 07:15:27.897517 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:27.897033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:42437)
0620 07:15:27.897345 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:27.897410 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":87,"threads_started":1}
I20260620 07:15:28.065495 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:28.065018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:56561)
0620 07:15:28.065334 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:28.065396 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260620 07:15:28.246423 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:28.245927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:33915)
0620 07:15:28.246246 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:28.246316 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260620 07:15:28.898939 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:28.898481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:33981)
0620 07:15:28.898771 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:28.898843 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
I20260620 07:15:29.066926 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:29.066367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:55229)
0620 07:15:29.066747 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:29.066821 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260620 07:15:29.247772 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:29.247252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:34407)
0620 07:15:29.247611 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:29.247672 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260620 07:15:29.900456 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:29.899900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:43251)
0620 07:15:29.900275 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:29.900350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":85,"threads_started":1}
I20260620 07:15:30.036705 21011 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:30.036851 21011 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:30.037358 21011 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
W20260620 07:15:30.038251 20600 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
W20260620 07:15:30.038349 20602 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
I20260620 07:15:30.038484 20602 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c635026273488a943c9e600acafb42; no voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e
I20260620 07:15:30.038678 21011 raft_consensus.cc:2726] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:15:30.068357 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:30.067792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:58357)
0620 07:15:30.068084 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:30.068184 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260620 07:15:30.249352 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:30.248809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:39371)
0620 07:15:30.249127 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:30.249204 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260620 07:15:30.901775 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:30.901350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:44659)
0620 07:15:30.901621 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:30.901680 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260620 07:15:31.069622 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:31.069190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:40351)
0620 07:15:31.069463 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:31.069525 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260620 07:15:31.250708 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:31.250203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:56163)
0620 07:15:31.250504 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:31.250564 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260620 07:15:31.903133 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:31.902722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:43861)
0620 07:15:31.902976 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:31.903036 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260620 07:15:32.070925 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:32.070465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:38375)
0620 07:15:32.070771 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:32.070829 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260620 07:15:32.252074 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:32.251588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:58829)
0620 07:15:32.251899 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:32.251967 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
I20260620 07:15:32.904732 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:32.904148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:47309)
0620 07:15:32.904469 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:32.904550 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260620 07:15:33.072597 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:33.071939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:44041)
0620 07:15:33.072377 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:33.072449 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":101,"threads_started":1}
I20260620 07:15:33.253504 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:33.253056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:50265)
0620 07:15:33.253340 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:33.253404 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260620 07:15:33.906262 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:33.905758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:56753)
0620 07:15:33.906076 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:33.906141 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260620 07:15:34.074052 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:34.073536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:38237)
0620 07:15:34.073886 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:34.073953 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":80,"threads_started":1}
I20260620 07:15:34.161128 21025 raft_consensus.cc:482] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:34.161335 21025 raft_consensus.cc:504] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f87febcbf54671bbbbc54c94a7518e" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45223 } } peers { permanent_uuid: "20c635026273488a943c9e600acafb42" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37045 } } peers { permanent_uuid: "a845699406eb4de3a8438a6ee4fb6969" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 44893 } }
I20260620 07:15:34.161859 21025 leader_election.cc:288] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223), a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893)
W20260620 07:15:34.162648 20602 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2f87febcbf54671bbbbc54c94a7518e (127.3.245.3:45223): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
W20260620 07:15:34.162902 20600 leader_election.cc:341] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a845699406eb4de3a8438a6ee4fb6969 (127.3.245.2:44893): Not found: Tablet not found: 188c4f0975c348679f7980863899551b
I20260620 07:15:34.162972 20600 leader_election.cc:302] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20c635026273488a943c9e600acafb42; no voters: a845699406eb4de3a8438a6ee4fb6969, b2f87febcbf54671bbbbc54c94a7518e
I20260620 07:15:34.163137 21025 raft_consensus.cc:2726] T 188c4f0975c348679f7980863899551b P 20c635026273488a943c9e600acafb42 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:15:34.254849 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:34.254387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:36067)
0620 07:15:34.254656 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:34.254726 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260620 07:15:34.907606 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:34.907172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:56853)
0620 07:15:34.907456 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:34.907512 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
I20260620 07:15:35.075342 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:35.074911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:37261)
0620 07:15:35.075182 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:35.075244 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260620 07:15:35.256245 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:35.255747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:35383)
0620 07:15:35.256065 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:35.256143 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":105,"threads_started":1}
I20260620 07:15:35.909176 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:35.908517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:40559)
0620 07:15:35.908931 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:35.909072 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":201,"threads_started":1}
I20260620 07:15:36.076786 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:36.076294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:40963)
0620 07:15:36.076587 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:36.076681 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260620 07:15:36.257861 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:36.257359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.3:40639)
0620 07:15:36.257659 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:36.257728 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260620 07:15:36.910769 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:36.910212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.1:53625)
0620 07:15:36.910590 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:36.910662 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":103,"threads_started":1}
I20260620 07:15:37.078312 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:37.077830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:45989 (local address 127.3.245.2:35531)
0620 07:15:37.078124 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:37.078188 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.62:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260620 07:15:37.238077 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20543
I20260620 07:15:37.244194 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20716
I20260620 07:15:37.250039 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 20850
2026-06-20T07:15:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59561 ms)
[----------] 30 tests from SecurityITest (248704 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:37 dist-test-slave-ldfr krb5kdc[21037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:37 dist-test-slave-ldfr krb5kdc[21037](info): set up 2 sockets
Jun 20 07:15:37 dist-test-slave-ldfr krb5kdc[21037](info): commencing operation
krb5kdc: starting...
W20260620 07:15:38.305017 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:38 dist-test-slave-ldfr krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:15:38.436717 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:43519
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:38.551746 21050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:38.552078 21050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:38.552170 21050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:38.556362 21050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:38.556488 21050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:38.556532 21050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:38.556600 21050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:38.556686 21050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:38.562297 21050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:38.563477 21050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:38.564916 21050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:38.574429 21058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:38.574498 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:38.574483 21050 server_base.cc:1034] running on GCE node
W20260620 07:15:38.574517 21055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:38.575479 21050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:38.576669 21050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:38.577860 21050 hybrid_clock.cc:648] HybridClock initialized: now 1781939738577828 us; error 45 us; skew 500 ppm
Jun 20 07:15:38 dist-test-slave-ldfr krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:38.580783 21050 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:15:38.581761 21050 webserver.cc:460] Webserver started at http://127.3.245.62:42221/ using document root <none> and password file <none>
I20260620 07:15:38.582284 21050 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:38.582342 21050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:38.582526 21050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:38.584730 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "166c1d5f00be4b3f8ad2ddae671d8def"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "68f11b1cd6067a30fcf3204ae7f42dd7"
server_key_iv: "bdf05d5584edddad3c7a0102cbb7c0be"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.585315 21050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "166c1d5f00be4b3f8ad2ddae671d8def"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "68f11b1cd6067a30fcf3204ae7f42dd7"
server_key_iv: "bdf05d5584edddad3c7a0102cbb7c0be"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.589259 21050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:15:38.591703 21065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:38.593176 21050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:38.593350 21050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "166c1d5f00be4b3f8ad2ddae671d8def"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "68f11b1cd6067a30fcf3204ae7f42dd7"
server_key_iv: "bdf05d5584edddad3c7a0102cbb7c0be"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.593485 21050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:38.609028 21050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:38.612449 21050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:38.612730 21050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:38.620193 21050 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:43519
I20260620 07:15:38.620208 21117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:43519 every 8 connection(s)
I20260620 07:15:38.621214 21050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:38.623375 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21050
I20260620 07:15:38.623533 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:38.623847 4052 external_mini_cluster.cc:1428] Setting key 42db3136fc2c501ad6d90a60cdde07fd
I20260620 07:15:38.624226 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:38.631266 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: Bootstrap starting.
Jun 20 07:15:38 dist-test-slave-ldfr krb5kdc[21037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:38.634104 21118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:38.635067 21118 log.cc:824] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:38.637379 21118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: No bootstrap required, opened a new log
I20260620 07:15:38.639447 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:38.625193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.3.245.62:43519)
0620 07:15:38.625608 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:38.625630 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:38.625665 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:15:38.626322 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:38.626338 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:38.626660 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:38.626930 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:38.627823 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.628921 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.630151 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.630629 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.630751 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:38.633646 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:38.633654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:38.633672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:38.633705 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:38.635765 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:38.636221 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:38.636226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:38.636233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:38.636373 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:38.636817 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:38.636820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:38.636822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:38.637342 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:38.637532 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:38.637939 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 07:15:38.638233 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":147,"threads_started":1}
I20260620 07:15:38.641275 21118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } }
I20260620 07:15:38.641531 21118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:38.641602 21118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166c1d5f00be4b3f8ad2ddae671d8def, State: Initialized, Role: FOLLOWER
I20260620 07:15:38.642084 21118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } }
I20260620 07:15:38.642215 21118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:38.642300 21118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:38.642393 21118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:38.643299 21118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } }
I20260620 07:15:38.643671 21118 leader_election.cc:302] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 166c1d5f00be4b3f8ad2ddae671d8def; no voters:
I20260620 07:15:38.644038 21118 leader_election.cc:288] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:38.644176 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:38.644529 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [term 1 LEADER]: Becoming Leader. State: Replica: 166c1d5f00be4b3f8ad2ddae671d8def, State: Running, Role: LEADER
I20260620 07:15:38.644888 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } }
I20260620 07:15:38.645359 21118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:38.646026 21123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [sys.catalog]: SysCatalogTable state changed. Reason: New leader 166c1d5f00be4b3f8ad2ddae671d8def. Latest consensus state: current_term: 1 leader_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } } }
I20260620 07:15:38.646237 21123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:38.646474 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "166c1d5f00be4b3f8ad2ddae671d8def" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43519 } } }
I20260620 07:15:38.646649 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:38.646953 21132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:38.650225 21132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:38.656445 21132 catalog_manager.cc:1347] Generated new cluster ID: de35d747d8d04faf99b6502b258dd143
I20260620 07:15:38.656524 21132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:38.670634 21132 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:38.671975 21132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:38.691295 21132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: Generated new TSK 0
I20260620 07:15:38.692126 21132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:15:38.732208 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:38.848487 21144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:38.848784 21144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:38.848862 21144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:38.852931 21144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:38.852998 21144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:38.853114 21144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:38.858508 21144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:38.859633 21144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:38.860772 21144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:38.871217 21152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:38.871253 21150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:38.871275 21149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:38.871379 21144 server_base.cc:1034] running on GCE node
I20260620 07:15:38.872023 21144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:38.872781 21144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:38.873976 21144 hybrid_clock.cc:648] HybridClock initialized: now 1781939738873947 us; error 43 us; skew 500 ppm
Jun 20 07:15:38 dist-test-slave-ldfr krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:38.877032 21144 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:15:38.877900 21144 webserver.cc:460] Webserver started at http://127.3.245.1:43883/ using document root <none> and password file <none>
I20260620 07:15:38.878501 21144 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:38.878561 21144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:38.878726 21144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:38.880810 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "f745e97543574b32be2bdad1da7279a7"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "109bc91be1b56a5520941d15cdc6ad14"
server_key_iv: "7b01f1c86cc0e1cec399da261b10933c"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.881345 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "f745e97543574b32be2bdad1da7279a7"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "109bc91be1b56a5520941d15cdc6ad14"
server_key_iv: "7b01f1c86cc0e1cec399da261b10933c"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.885192 21144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:15:38.887759 21159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:38.889400 21144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:38.889521 21144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "f745e97543574b32be2bdad1da7279a7"
format_stamp: "Formatted at 2026-06-20 07:15:38 on dist-test-slave-ldfr"
server_key: "109bc91be1b56a5520941d15cdc6ad14"
server_key_iv: "7b01f1c86cc0e1cec399da261b10933c"
server_key_version: "encryptionkey@0"
I20260620 07:15:38.889648 21144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:38.906708 21144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:38.910252 21144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:38.910486 21144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:38.911115 21144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:38.911963 21144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:38.912039 21144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:38.912108 21144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:38.912160 21144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:38.921059 21144 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:37219
I20260620 07:15:38.921125 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:37219 every 8 connection(s)
I20260620 07:15:38.922019 21144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:38.929242 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21144
I20260620 07:15:38.929388 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:38.929713 4052 external_mini_cluster.cc:1428] Setting key 3ab1e331cb9f407f0abe373fe7ec873e
Jun 20 07:15:38 dist-test-slave-ldfr krb5kdc[21037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939738, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:38.936316 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:38.923949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:56227 (local address 127.3.245.62:43519)
0620 07:15:38.924097 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:38.924101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:38.924917 (+ 816us) server_negotiation.cc:407] Connection header received
0620 07:15:38.925831 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:38.925835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:38.925896 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:38.926008 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:38.927220 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.927769 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.929030 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.929267 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.929397 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:38.932287 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:38.932291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:38.932294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:38.932322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:38.934145 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:38.934691 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:38.934694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:38.934696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:38.934783 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:38.935131 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:38.935136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:38.935138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:38.935393 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:38.935486 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:38.936002 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 07:15:38.936139 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:15:38.937503 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:38.924234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43519 (local address 127.3.245.1:56227)
0620 07:15:38.924753 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:38.924805 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:38.925617 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:38.926184 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:38.926192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:38.926581 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:38.927060 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:38.927073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.927928 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.927931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:38.928890 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:38.928900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:38.929475 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:38.929481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:38.929700 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:38.930469 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:38.930489 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:38.932090 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:38.934284 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:38.934290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:38.934307 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:38.934572 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:38.934879 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:38.934883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:38.934885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:38.935030 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:38.935487 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:38.935494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:38.935756 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 07:15:38.935962 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 07:15:38.936308 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260620 07:15:38.939111 21273 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43519
I20260620 07:15:38.939389 21273 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:15:38.939970 21273 heartbeater.cc:511] Master 127.3.245.62:43519 requested a full tablet report, sending...
I20260620 07:15:38.941735 21082 ts_manager.cc:194] Registered new tserver with Master: f745e97543574b32be2bdad1da7279a7 (127.3.245.1:37219)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:15:38.943122 21082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:56227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:15:38.961427 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:39.077399 21278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:39.077747 21278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:39.077796 21278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:39.081951 21278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:39.082026 21278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:39.082087 21278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:39.087605 21278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:39.088708 21278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:39.089690 21278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:39.100111 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:39.100232 21278 server_base.cc:1034] running on GCE node
W20260620 07:15:39.100180 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:39.100471 21286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:39.100925 21278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:39.101711 21278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:39.102934 21278 hybrid_clock.cc:648] HybridClock initialized: now 1781939739102901 us; error 52 us; skew 500 ppm
Jun 20 07:15:39 dist-test-slave-ldfr krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:39.105837 21278 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:15:39.106707 21278 webserver.cc:460] Webserver started at http://127.3.245.2:41579/ using document root <none> and password file <none>
I20260620 07:15:39.107236 21278 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:39.107303 21278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:39.107466 21278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:39.109555 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "c07e22e701c84cb7aab7c2e09e4af927"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "3d67fa0031d1947aade3df9c7bde9629"
server_key_iv: "91532d035a984d1790c8041be26b5e97"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.110067 21278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "c07e22e701c84cb7aab7c2e09e4af927"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "3d67fa0031d1947aade3df9c7bde9629"
server_key_iv: "91532d035a984d1790c8041be26b5e97"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.113823 21278 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 07:15:39.116219 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.117774 21278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:15:39.117882 21278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "c07e22e701c84cb7aab7c2e09e4af927"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "3d67fa0031d1947aade3df9c7bde9629"
server_key_iv: "91532d035a984d1790c8041be26b5e97"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.118129 21278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:39.152743 21278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:39.155982 21278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:39.156217 21278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:39.156927 21278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:39.157793 21278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:39.157842 21278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.157881 21278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:39.157938 21278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.166858 21278 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:46209
I20260620 07:15:39.166879 21406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:46209 every 8 connection(s)
I20260620 07:15:39.167757 21278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:39.168983 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21278
I20260620 07:15:39.169099 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:39.169411 4052 external_mini_cluster.cc:1428] Setting key 174dd02a1bfbbe5087c9f5b651f4bc03
Jun 20 07:15:39 dist-test-slave-ldfr krb5kdc[21037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:15:39.186175 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.170065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:43903 (local address 127.3.245.62:43519)
0620 07:15:39.170214 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.170221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.171361 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 07:15:39.172793 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.172800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.172870 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.172983 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:39.174708 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.175663 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.177469 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.177841 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.177969 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.181489 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:39.181495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:39.181500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:39.181539 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:39.183386 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.184133 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.184136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.184137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.184220 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.184595 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.184598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.184600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.185020 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:39.185120 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.185845 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.185974 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:15:39.187005 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.170575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43519 (local address 127.3.245.2:43903)
0620 07:15:39.171188 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.171264 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.172530 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.173186 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.173196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.173743 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:39.174476 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.174499 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.175812 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.175815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.177351 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.177362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.178012 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.178015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.178262 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.179025 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:39.179046 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:39.181277 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:39.183512 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:39.183517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:39.183531 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:39.183990 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:39.184314 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:39.184316 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:39.184318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:39.184461 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:39.185141 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:39.185147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:39.185461 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.185685 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.186106 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":202,"threads_started":1}
I20260620 07:15:39.188345 21407 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43519
I20260620 07:15:39.188585 21407 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:39.189131 21407 heartbeater.cc:511] Master 127.3.245.62:43519 requested a full tablet report, sending...
I20260620 07:15:39.190214 21082 ts_manager.cc:194] Registered new tserver with Master: c07e22e701c84cb7aab7c2e09e4af927 (127.3.245.2:46209)
I20260620 07:15:39.190881 21082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:43903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:15:39.203428 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:39.322134 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:39.322420 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:39.322465 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:39.327188 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:39.327255 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:39.327316 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:39.336203 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:39.337762 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:39.339207 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:39.350925 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:39.351467 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:39.351771 21412 server_base.cc:1034] running on GCE node
W20260620 07:15:39.353637 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:39.354163 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:39.355020 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:39.356287 21412 hybrid_clock.cc:648] HybridClock initialized: now 1781939739356241 us; error 52 us; skew 500 ppm
Jun 20 07:15:39 dist-test-slave-ldfr krb5kdc[21037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:39.360324 21412 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:15:39.361423 21412 webserver.cc:460] Webserver started at http://127.3.245.3:46355/ using document root <none> and password file <none>
I20260620 07:15:39.362183 21412 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:39.362272 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:39.362493 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:39.365233 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "7d77716129ec4b46a0993940c7ce840c"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "29bd7efb5e82e0f333a834f3fedac446"
server_key_iv: "e93eb0a0d9c881699c9e71b06b442e21"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.366011 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "7d77716129ec4b46a0993940c7ce840c"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "29bd7efb5e82e0f333a834f3fedac446"
server_key_iv: "e93eb0a0d9c881699c9e71b06b442e21"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.371518 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:15:39.374895 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.376506 21412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:39.376677 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "7d77716129ec4b46a0993940c7ce840c"
format_stamp: "Formatted at 2026-06-20 07:15:39 on dist-test-slave-ldfr"
server_key: "29bd7efb5e82e0f333a834f3fedac446"
server_key_iv: "e93eb0a0d9c881699c9e71b06b442e21"
server_key_version: "encryptionkey@0"
I20260620 07:15:39.376873 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:39.397212 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:39.400391 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:39.400543 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:39.401168 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:39.401952 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:39.401999 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.402040 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:39.402055 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:39.411060 21412 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:33383
I20260620 07:15:39.411083 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:33383 every 8 connection(s)
I20260620 07:15:39.411962 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
Jun 20 07:15:39 dist-test-slave-ldfr krb5kdc[21037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939739, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:39.420991 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21412
I20260620 07:15:39.421139 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:39.421478 4052 external_mini_cluster.cc:1428] Setting key 039754d174a8cad919821ed9d4f0ee6c
I20260620 07:15:39.425582 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.413797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:45891 (local address 127.3.245.62:43519)
0620 07:15:39.413978 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.413983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.414797 (+ 814us) server_negotiation.cc:407] Connection header received
0620 07:15:39.415609 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.415612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.415658 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.415730 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:39.416974 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.417499 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.418624 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.418857 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.418936 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.421487 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:39.421495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:39.421498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:39.421522 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:39.423337 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.423973 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.423977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.423979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.424068 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.424427 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.424430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.424432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.424674 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:39.424794 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.425301 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.425429 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 07:15:39.426558 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.414181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43519 (local address 127.3.245.3:45891)
0620 07:15:39.414625 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.414666 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.415414 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.415893 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.415899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.416237 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:39.416835 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.416852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.417615 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.417618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.418486 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.418495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.418989 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.418992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.419131 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.419681 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:39.419696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:39.421321 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:39.423459 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:39.423465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:39.423480 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:39.423867 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:39.424169 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:39.424172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:39.424174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:39.424321 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:39.424794 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:39.424802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:39.425066 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.425267 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.425589 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260620 07:15:39.427747 21541 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43519
I20260620 07:15:39.428023 21541 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:39.428503 21541 heartbeater.cc:511] Master 127.3.245.62:43519 requested a full tablet report, sending...
I20260620 07:15:39.429519 21082 ts_manager.cc:194] Registered new tserver with Master: 7d77716129ec4b46a0993940c7ce840c (127.3.245.3:33383)
I20260620 07:15:39.430081 21082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:45891
I20260620 07:15:39.435401 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:39.444420 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.437349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.3.245.62:43519)
0620 07:15:39.437468 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.437471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.437592 (+ 121us) server_negotiation.cc:407] Connection header received
0620 07:15:39.437687 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.437689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.437726 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.437820 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:39.438561 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.439080 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.439928 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.440134 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.440211 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.441579 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:39.441582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:39.441584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:39.441604 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:39.443021 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.443408 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.443410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.443411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.443480 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:39.443710 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:39.443712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:39.443714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:39.443930 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:39.444030 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.444223 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.444311 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:15:39.447321 21082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:39.450629 21082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:39.466358 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.459711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37219 (local address 127.0.0.1:40374)
0620 07:15:39.460514 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.460571 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.460742 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.461523 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.461529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.461552 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:39.461697 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.461707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463489 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.463493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.464740 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.464750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.465738 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.465741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.465862 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.465905 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.466028 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.466162 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":123,"spinlock_wait_cycles":6016,"thread_start_us":86,"threads_started":1}
I20260620 07:15:39.466398 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.459936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.3.245.1:37219)
0620 07:15:39.461033 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.461041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.461064 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:15:39.461154 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.461159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.461365 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.461508 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:39.462055 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463339 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.464872 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.465601 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.465722 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.465882 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.466080 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.466208 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":95,"threads_started":1}
I20260620 07:15:39.466485 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.460473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:33383 (local address 127.0.0.1:37930)
0620 07:15:39.461024 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.461037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.461114 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.461762 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.461765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.461792 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:39.461904 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.461913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463800 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.463805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.465156 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.465167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.466108 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.466111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.466187 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.466206 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.466253 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.466317 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":126,"spinlock_wait_cycles":5888,"thread_start_us":87,"threads_started":1}
I20260620 07:15:39.467039 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.460869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.3.245.3:33383)
0620 07:15:39.461331 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.461339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.461356 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:15:39.461412 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.461416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.461616 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.461870 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:39.462111 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463611 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.465328 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.465983 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.466586 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.466739 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.466792 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.466892 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":91,"threads_started":1}
I20260620 07:15:39.467344 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.459905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:46209 (local address 127.0.0.1:42874)
0620 07:15:39.460515 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.460573 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.460741 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.461762 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.461765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.461792 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:39.462078 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.462097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463489 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.463493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.465156 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.465167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.466299 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.466303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.466394 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.466430 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.466959 (+ 529us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.467036 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":306,"spinlock_wait_cycles":11136,"thread_start_us":111,"threads_started":1}
I20260620 07:15:39.468104 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.460139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.3.245.2:46209)
0620 07:15:39.461266 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.461276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.461296 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:15:39.461361 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.461365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.461517 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.461864 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:39.462215 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.463338 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.465294 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.466145 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.467642 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.467816 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.467868 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.467956 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":94,"threads_started":1}
I20260620 07:15:39.469779 21475 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee314e9b9274f5bb826482e944e1347 (DEFAULT_TABLE table=test-table [id=256f132046fe4f848cf757d6213490ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:39.469835 21207 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee314e9b9274f5bb826482e944e1347 (DEFAULT_TABLE table=test-table [id=256f132046fe4f848cf757d6213490ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:39.470772 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee314e9b9274f5bb826482e944e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:39.470844 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee314e9b9274f5bb826482e944e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:39.470701 21341 tablet_service.cc:1467] Processing CreateTablet for tablet 1ee314e9b9274f5bb826482e944e1347 (DEFAULT_TABLE table=test-table [id=256f132046fe4f848cf757d6213490ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:39.471590 21341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ee314e9b9274f5bb826482e944e1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:39.477249 21558 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Bootstrap starting.
I20260620 07:15:39.477914 21559 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Bootstrap starting.
I20260620 07:15:39.478947 21560 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Bootstrap starting.
I20260620 07:15:39.479882 21558 tablet_bootstrap.cc:654] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:39.480317 21559 tablet_bootstrap.cc:654] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:39.481155 21560 tablet_bootstrap.cc:654] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:39.481230 21559 log.cc:824] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:39.481230 21558 log.cc:824] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:39.482039 21560 log.cc:824] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:39.483397 21558 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: No bootstrap required, opened a new log
I20260620 07:15:39.483393 21559 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: No bootstrap required, opened a new log
I20260620 07:15:39.483600 21558 ts_tablet_manager.cc:1397] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:15:39.483592 21559 ts_tablet_manager.cc:1397] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:15:39.484472 21560 tablet_bootstrap.cc:492] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: No bootstrap required, opened a new log
I20260620 07:15:39.484694 21560 ts_tablet_manager.cc:1397] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:15:39.487452 21560 raft_consensus.cc:348] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.487715 21560 raft_consensus.cc:374] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:39.487773 21560 raft_consensus.cc:729] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07e22e701c84cb7aab7c2e09e4af927, State: Initialized, Role: FOLLOWER
I20260620 07:15:39.487730 21558 raft_consensus.cc:348] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.488013 21558 raft_consensus.cc:374] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:39.488062 21558 raft_consensus.cc:729] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d77716129ec4b46a0993940c7ce840c, State: Initialized, Role: FOLLOWER
I20260620 07:15:39.488135 21559 raft_consensus.cc:348] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.488361 21559 raft_consensus.cc:374] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:39.488246 21560 consensus_queue.cc:260] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.488425 21559 raft_consensus.cc:729] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f745e97543574b32be2bdad1da7279a7, State: Initialized, Role: FOLLOWER
I20260620 07:15:39.488502 21558 consensus_queue.cc:260] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.488839 21559 consensus_queue.cc:260] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.489176 21407 heartbeater.cc:503] Master 127.3.245.62:43519 was elected leader, sending a full tablet report...
I20260620 07:15:39.489399 21560 ts_tablet_manager.cc:1428] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:15:39.489457 21558 ts_tablet_manager.cc:1428] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:15:39.489647 21559 ts_tablet_manager.cc:1428] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 07:15:39.489811 21541 heartbeater.cc:503] Master 127.3.245.62:43519 was elected leader, sending a full tablet report...
I20260620 07:15:39.489943 21273 heartbeater.cc:503] Master 127.3.245.62:43519 was elected leader, sending a full tablet report...
W20260620 07:15:39.662973 21542 tablet.cc:2369] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:39.668946 21408 tablet.cc:2369] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:39.673450 21274 tablet.cc:2369] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:39.798503 21565 raft_consensus.cc:482] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:39.798709 21565 raft_consensus.cc:504] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.799901 21565 leader_election.cc:288] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f745e97543574b32be2bdad1da7279a7 (127.3.245.1:37219), c07e22e701c84cb7aab7c2e09e4af927 (127.3.245.2:46209)
I20260620 07:15:39.803540 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.800102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37219 (local address 127.3.245.3:40497)
0620 07:15:39.800257 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.800274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.800377 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.800712 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.800715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.800735 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:39.800859 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.800865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.801619 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.801625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.802400 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.802409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.803084 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.803087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.803199 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.803218 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.803325 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.803397 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:15:39.803540 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.800273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:40497 (local address 127.3.245.1:37219)
0620 07:15:39.800405 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.800410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.800430 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:15:39.800491 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.800494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.800545 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.800694 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:39.800983 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.801503 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.802530 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.802962 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.803084 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.803151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.803345 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.803418 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:15:39.804008 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.800276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:46209 (local address 127.3.245.3:53317)
0620 07:15:39.800712 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.800724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:39.800838 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:39.801096 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:39.801099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:39.801114 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:39.801237 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.801243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.801949 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.801954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.802733 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:39.802741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.803399 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.803403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:39.803511 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.803535 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 07:15:39.803781 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 07:15:39.803858 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260620 07:15:39.804045 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.800418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:53317 (local address 127.3.245.2:46209)
0620 07:15:39.800530 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.800535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.800812 (+ 277us) server_negotiation.cc:407] Connection header received
0620 07:15:39.800945 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.800948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.800999 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.801084 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:39.801351 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.801835 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.802862 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.803256 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.803396 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.803452 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.803857 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.803931 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:15:39.804296 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee314e9b9274f5bb826482e944e1347" candidate_uuid: "7d77716129ec4b46a0993940c7ce840c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f745e97543574b32be2bdad1da7279a7" is_pre_election: true
I20260620 07:15:39.804663 21227 raft_consensus.cc:2454] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d77716129ec4b46a0993940c7ce840c in term 0.
I20260620 07:15:39.804664 21361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee314e9b9274f5bb826482e944e1347" candidate_uuid: "7d77716129ec4b46a0993940c7ce840c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e22e701c84cb7aab7c2e09e4af927" is_pre_election: true
I20260620 07:15:39.804929 21361 raft_consensus.cc:2454] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d77716129ec4b46a0993940c7ce840c in term 0.
I20260620 07:15:39.805181 21431 leader_election.cc:302] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d77716129ec4b46a0993940c7ce840c, f745e97543574b32be2bdad1da7279a7; no voters:
I20260620 07:15:39.805511 21565 raft_consensus.cc:2781] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:39.805600 21565 raft_consensus.cc:482] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:39.805646 21565 raft_consensus.cc:3037] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:39.806659 21565 raft_consensus.cc:504] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.806981 21565 leader_election.cc:288] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [CANDIDATE]: Term 1 election: Requested vote from peers f745e97543574b32be2bdad1da7279a7 (127.3.245.1:37219), c07e22e701c84cb7aab7c2e09e4af927 (127.3.245.2:46209)
I20260620 07:15:39.807305 21227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee314e9b9274f5bb826482e944e1347" candidate_uuid: "7d77716129ec4b46a0993940c7ce840c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f745e97543574b32be2bdad1da7279a7"
I20260620 07:15:39.807404 21361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ee314e9b9274f5bb826482e944e1347" candidate_uuid: "7d77716129ec4b46a0993940c7ce840c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e22e701c84cb7aab7c2e09e4af927"
I20260620 07:15:39.807423 21227 raft_consensus.cc:3037] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:39.807579 21361 raft_consensus.cc:3037] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:39.808555 21227 raft_consensus.cc:2454] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d77716129ec4b46a0993940c7ce840c in term 1.
I20260620 07:15:39.808669 21361 raft_consensus.cc:2454] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d77716129ec4b46a0993940c7ce840c in term 1.
I20260620 07:15:39.809005 21430 leader_election.cc:302] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d77716129ec4b46a0993940c7ce840c, c07e22e701c84cb7aab7c2e09e4af927; no voters:
I20260620 07:15:39.809219 21565 raft_consensus.cc:2781] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:39.809458 21565 raft_consensus.cc:686] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 1 LEADER]: Becoming Leader. State: Replica: 7d77716129ec4b46a0993940c7ce840c, State: Running, Role: LEADER
I20260620 07:15:39.809808 21565 consensus_queue.cc:237] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } }
I20260620 07:15:39.813165 21080 catalog_manager.cc:5495] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d77716129ec4b46a0993940c7ce840c (127.3.245.3). New cstate: current_term: 1 leader_uuid: "7d77716129ec4b46a0993940c7ce840c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d77716129ec4b46a0993940c7ce840c" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 33383 } health_report { overall_health: HEALTHY } } }
I20260620 07:15:39.831223 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.827526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.3.245.3:33383)
0620 07:15:39.827750 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:39.827756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:39.827769 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:39.827846 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:39.827849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:39.827900 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:39.828015 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:39.828303 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.828846 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.829712 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:39.829937 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:39.830050 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:39.830378 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:39.830802 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:39.830934 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:39.830998 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 07:15:39.831055 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:15:39.837502 21227 raft_consensus.cc:1261] T 1ee314e9b9274f5bb826482e944e1347 P f745e97543574b32be2bdad1da7279a7 [term 1 FOLLOWER]: Refusing update from remote peer 7d77716129ec4b46a0993940c7ce840c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:39.837502 21361 raft_consensus.cc:1261] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 1 FOLLOWER]: Refusing update from remote peer 7d77716129ec4b46a0993940c7ce840c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:39.838264 21565 consensus_queue.cc:1035] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07e22e701c84cb7aab7c2e09e4af927" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 46209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:39.838439 21568 consensus_queue.cc:1035] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f745e97543574b32be2bdad1da7279a7" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 37219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:39.849936 21572 mvcc.cc:205] Tried to move back new op lower bound from 7298825174365503488 to 7298825174263394304. Current Snapshot: MvccSnapshot[applied={T|T < 7298825174365503488}]
I20260620 07:15:39.854781 21081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:39.854941 21081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:39.857486 21081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 166c1d5f00be4b3f8ad2ddae671d8def: Sending DeleteTablet for 3 replicas of tablet 1ee314e9b9274f5bb826482e944e1347
I20260620 07:15:39.858317 21341 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee314e9b9274f5bb826482e944e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:39 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:42874
I20260620 07:15:39.858552 21207 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee314e9b9274f5bb826482e944e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:39 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:40374
I20260620 07:15:39.858860 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21144
I20260620 07:15:39.859022 21475 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ee314e9b9274f5bb826482e944e1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:39 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:37930
I20260620 07:15:39.864893 21581 tablet_replica.cc:330] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927: stopping tablet replica
I20260620 07:15:39.865253 21583 tablet_replica.cc:330] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: stopping tablet replica
I20260620 07:15:39.865309 21581 raft_consensus.cc:2229] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:39.865674 21583 raft_consensus.cc:2229] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:39.865713 21581 raft_consensus.cc:2258] T 1ee314e9b9274f5bb826482e944e1347 P c07e22e701c84cb7aab7c2e09e4af927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:39.866156 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21278
W20260620 07:15:39.866130 21069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:37219 (error 108)
I20260620 07:15:39.867060 21583 raft_consensus.cc:2258] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:39.867672 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.867273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:37219 (local address 127.0.0.1:40388)
0620 07:15:39.867486 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.867564 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 07:15:39.867877 21069 catalog_manager.cc:4584] TS f745e97543574b32be2bdad1da7279a7 (127.3.245.1:37219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ee314e9b9274f5bb826482e944e1347: Network error: Client connection negotiation failed: client connection to 127.3.245.1:37219: connect: Connection refused (error 111)
I20260620 07:15:39.868886 21583 ts_tablet_manager.cc:1905] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:39.871726 21583 ts_tablet_manager.cc:1918] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:39.871835 21583 log.cc:1196] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/1ee314e9b9274f5bb826482e944e1347
I20260620 07:15:39.872165 21583 ts_tablet_manager.cc:1939] T 1ee314e9b9274f5bb826482e944e1347 P 7d77716129ec4b46a0993940c7ce840c: Deleting consensus metadata
I20260620 07:15:39.873478 21069 catalog_manager.cc:4841] TS 7d77716129ec4b46a0993940c7ce840c (127.3.245.3:33383): tablet 1ee314e9b9274f5bb826482e944e1347 (table test-table [id=256f132046fe4f848cf757d6213490ca]) successfully deleted
I20260620 07:15:39.873518 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21412
I20260620 07:15:39.874303 21553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:39.873673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:46209 (local address 127.0.0.1:42890)
0620 07:15:39.874129 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:39.874200 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:46209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
I20260620 07:15:39.879149 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21050
2026-06-20T07:15:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2629 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T07:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:39Z Disabled control of system clock
I20260620 07:15:39.911360 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36711
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:40.025135 21588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:40.025419 21588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:40.025456 21588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:40.029539 21588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:40.029613 21588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:40.029628 21588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:40.029645 21588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:40.029660 21588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:40.034534 21588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:40.035586 21588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:40.036943 21588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:40.045637 21596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.045686 21588 server_base.cc:1034] running on GCE node
W20260620 07:15:40.045632 21593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.045629 21594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.046491 21588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:40.047629 21588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:40.048815 21588 hybrid_clock.cc:648] HybridClock initialized: now 1781939740048801 us; error 26 us; skew 500 ppm
I20260620 07:15:40.050861 21588 webserver.cc:460] Webserver started at http://127.3.245.62:45747/ using document root <none> and password file <none>
I20260620 07:15:40.051460 21588 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:40.051522 21588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:40.051748 21588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:40.053896 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "de596df6eac04ba090c19761a6ea0c32"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "62ddfdfc20ef0e3c41f3dc5d603c0a21"
server_key_iv: "40671b8d31fe7866e24728f326df4896"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.054478 21588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "de596df6eac04ba090c19761a6ea0c32"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "62ddfdfc20ef0e3c41f3dc5d603c0a21"
server_key_iv: "40671b8d31fe7866e24728f326df4896"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.058552 21588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:15:40.061456 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.062768 21588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:40.062934 21588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "de596df6eac04ba090c19761a6ea0c32"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "62ddfdfc20ef0e3c41f3dc5d603c0a21"
server_key_iv: "40671b8d31fe7866e24728f326df4896"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.063053 21588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:40.093779 21588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:40.094542 21588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:40.094760 21588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:40.102306 21588 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:36711
I20260620 07:15:40.102353 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:36711 every 8 connection(s)
I20260620 07:15:40.103355 21588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:40.106797 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:40.107736 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21588
I20260620 07:15:40.107879 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:40.108206 4052 external_mini_cluster.cc:1428] Setting key 48f7d7d60ac524166bd9f6774a16200b
I20260620 07:15:40.113229 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: Bootstrap starting.
I20260620 07:15:40.115976 21655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:40.116951 21655 log.cc:824] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:40.118692 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.109753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.3.245.62:36711)
0620 07:15:40.110427 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.110444 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.110481 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:15:40.111196 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.111213 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.111615 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.111905 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:40.112475 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.113572 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.114908 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.115435 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.115584 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.116569 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:40.116574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:40.116589 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:40.117178 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:40.117240 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:40.117264 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:40.117361 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.117582 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.117889 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":149,"threads_started":1}
I20260620 07:15:40.119415 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: No bootstrap required, opened a new log
I20260620 07:15:40.123837 21655 raft_consensus.cc:348] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } }
I20260620 07:15:40.124223 21655 raft_consensus.cc:374] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:40.124330 21655 raft_consensus.cc:729] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de596df6eac04ba090c19761a6ea0c32, State: Initialized, Role: FOLLOWER
I20260620 07:15:40.124883 21655 consensus_queue.cc:260] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } }
I20260620 07:15:40.125077 21655 raft_consensus.cc:388] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:40.125152 21655 raft_consensus.cc:482] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:40.125293 21655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:40.126341 21655 raft_consensus.cc:504] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } }
I20260620 07:15:40.126709 21655 leader_election.cc:302] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de596df6eac04ba090c19761a6ea0c32; no voters:
I20260620 07:15:40.127012 21655 leader_election.cc:288] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:40.127126 21660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:40.127369 21660 raft_consensus.cc:686] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [term 1 LEADER]: Becoming Leader. State: Replica: de596df6eac04ba090c19761a6ea0c32, State: Running, Role: LEADER
I20260620 07:15:40.127682 21660 consensus_queue.cc:237] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } }
I20260620 07:15:40.128470 21655 sys_catalog.cc:564] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:40.129752 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de596df6eac04ba090c19761a6ea0c32. Latest consensus state: current_term: 1 leader_uuid: "de596df6eac04ba090c19761a6ea0c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } } }
I20260620 07:15:40.129750 21661 sys_catalog.cc:455] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de596df6eac04ba090c19761a6ea0c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de596df6eac04ba090c19761a6ea0c32" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36711 } } }
I20260620 07:15:40.129947 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:40.129952 21661 sys_catalog.cc:458] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:40.130496 21669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:40.133811 21669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:40.139757 21669 catalog_manager.cc:1347] Generated new cluster ID: 5015ca0335c04a76a3b05562709db4a7
I20260620 07:15:40.139843 21669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:40.152261 21669 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:40.153590 21669 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:40.168185 21669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: Generated new TSK 0
I20260620 07:15:40.169142 21669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:40.174547 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--builtin_ntp_servers=127.3.245.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:40.289731 21679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:40.290050 21679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:40.290122 21679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:40.294317 21679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:40.294384 21679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:40.294441 21679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:40.299253 21679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:40.300441 21679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:40.301852 21679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:40.311590 21687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.311590 21685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.311640 21684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.312345 21679 server_base.cc:1034] running on GCE node
I20260620 07:15:40.312817 21679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:40.313460 21679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:40.314668 21679 hybrid_clock.cc:648] HybridClock initialized: now 1781939740314657 us; error 31 us; skew 500 ppm
I20260620 07:15:40.316747 21679 webserver.cc:460] Webserver started at http://127.3.245.1:46833/ using document root <none> and password file <none>
I20260620 07:15:40.317342 21679 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:40.317412 21679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:40.317624 21679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:40.319727 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "ae9439e6726d4443855c20513ab3a5b3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "44ad451f8a748feb9ff0d57be0905801"
server_key_iv: "4b7dfcca7970416b8f17cd74ea93fd0d"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.320293 21679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9439e6726d4443855c20513ab3a5b3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "44ad451f8a748feb9ff0d57be0905801"
server_key_iv: "4b7dfcca7970416b8f17cd74ea93fd0d"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.324092 21679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 07:15:40.326694 21693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.327888 21679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:40.328028 21679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "ae9439e6726d4443855c20513ab3a5b3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "44ad451f8a748feb9ff0d57be0905801"
server_key_iv: "4b7dfcca7970416b8f17cd74ea93fd0d"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.328207 21679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:40.348043 21679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:40.348932 21679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:40.349144 21679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:40.349797 21679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:40.350791 21679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:40.350862 21679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.350948 21679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:40.350997 21679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.359997 21679 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:34695
I20260620 07:15:40.360061 21806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:34695 every 8 connection(s)
I20260620 07:15:40.361023 21679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:40.361922 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21679
I20260620 07:15:40.362033 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:40.362346 4052 external_mini_cluster.cc:1428] Setting key 6e876f35a05ea5c1b5daff51caba722b
I20260620 07:15:40.364681 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--builtin_ntp_servers=127.3.245.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:40.372035 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.363132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:48841 (local address 127.3.245.62:36711)
0620 07:15:40.363292 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.363296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.364547 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 07:15:40.366198 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.366201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.366273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.366370 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:40.367650 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.368563 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.369742 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.370165 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.370297 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.371434 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:40.371441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:40.371447 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:40.371463 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:40.371488 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:40.371502 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:40.371591 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.371741 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.371817 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:15:40.372926 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.363709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36711 (local address 127.3.245.1:48841)
0620 07:15:40.364407 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.364447 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.365937 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.366592 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.366603 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.366726 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:40.367491 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.367507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.368721 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.368724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.369612 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.369621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.370346 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.370349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.370488 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.371144 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:40.371145 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:40.371185 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:40.371248 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:40.371594 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:40.371601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:40.371605 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.371699 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.371923 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260620 07:15:40.374321 21807 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36711
I20260620 07:15:40.374567 21807 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:40.375092 21807 heartbeater.cc:511] Master 127.3.245.62:36711 requested a full tablet report, sending...
I20260620 07:15:40.376742 21619 ts_manager.cc:194] Registered new tserver with Master: ae9439e6726d4443855c20513ab3a5b3 (127.3.245.1:34695)
I20260620 07:15:40.378268 21619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.1:48841
W20260620 07:15:40.477519 21810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:40.477803 21810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:40.477842 21810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:40.482213 21810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:40.482290 21810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:40.482349 21810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:40.487242 21810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:40.488324 21810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:40.489699 21810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:40.499425 21818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.499475 21816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.499579 21815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.499552 21810 server_base.cc:1034] running on GCE node
I20260620 07:15:40.500190 21810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:40.500960 21810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:40.502157 21810 hybrid_clock.cc:648] HybridClock initialized: now 1781939740502143 us; error 29 us; skew 500 ppm
I20260620 07:15:40.504029 21810 webserver.cc:460] Webserver started at http://127.3.245.2:38873/ using document root <none> and password file <none>
I20260620 07:15:40.504689 21810 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:40.504783 21810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:40.505020 21810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:40.507158 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "de151ad246c94edb9ee8a8796d0867c3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "67e6c2b29137731a997df37ef5706b89"
server_key_iv: "11554860092a5c00bf1f607830216f9b"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.507810 21810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "de151ad246c94edb9ee8a8796d0867c3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "67e6c2b29137731a997df37ef5706b89"
server_key_iv: "11554860092a5c00bf1f607830216f9b"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.511848 21810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:15:40.514945 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.516686 21810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:40.516813 21810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "de151ad246c94edb9ee8a8796d0867c3"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "67e6c2b29137731a997df37ef5706b89"
server_key_iv: "11554860092a5c00bf1f607830216f9b"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.516943 21810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:40.544256 21810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:40.545032 21810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:40.545195 21810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:40.545754 21810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:40.546691 21810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:40.546737 21810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.546777 21810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:40.546793 21810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.555482 21810 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:42405
I20260620 07:15:40.555505 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:42405 every 8 connection(s)
I20260620 07:15:40.556447 21810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:40.561906 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21810
I20260620 07:15:40.562052 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:40.562346 4052 external_mini_cluster.cc:1428] Setting key 4dcce898bb1d5930b357d954df5a41a3
I20260620 07:15:40.564345 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--builtin_ntp_servers=127.3.245.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:40.566771 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.558431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:54181 (local address 127.3.245.62:36711)
0620 07:15:40.558576 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.558579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.559638 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 07:15:40.561328 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.561331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.561395 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.561492 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:40.562582 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.563215 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.564514 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.564878 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.565001 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.566158 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:40.566164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:40.566169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:40.566185 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:40.566212 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:40.566226 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:40.566314 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.566476 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.566551 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:40.567611 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.558817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36711 (local address 127.3.245.2:54181)
0620 07:15:40.559477 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.559518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.561085 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.561694 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.561701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.561797 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:40.562366 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.562381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.563350 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.563353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.564384 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.564394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.565043 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.565048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.565216 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.565883 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:40.565884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:40.565927 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:40.565999 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:40.566338 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:40.566345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:40.566349 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.566448 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.566748 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":104,"threads_started":1}
I20260620 07:15:40.568918 21938 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36711
I20260620 07:15:40.569159 21938 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:40.569697 21938 heartbeater.cc:511] Master 127.3.245.62:36711 requested a full tablet report, sending...
I20260620 07:15:40.570737 21619 ts_manager.cc:194] Registered new tserver with Master: de151ad246c94edb9ee8a8796d0867c3 (127.3.245.2:42405)
I20260620 07:15:40.571327 21619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.2:54181
W20260620 07:15:40.677791 21941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:40.678061 21941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:40.678122 21941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:40.682593 21941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:40.682667 21941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:40.682724 21941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:40.687765 21941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:40.688927 21941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:40.690363 21941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:40.700292 21949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:40.700351 21947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.700409 21941 server_base.cc:1034] running on GCE node
W20260620 07:15:40.700259 21946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:40.700919 21941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:40.701562 21941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:40.702719 21941 hybrid_clock.cc:648] HybridClock initialized: now 1781939740702701 us; error 30 us; skew 500 ppm
I20260620 07:15:40.704730 21941 webserver.cc:460] Webserver started at http://127.3.245.3:39831/ using document root <none> and password file <none>
I20260620 07:15:40.705325 21941 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:40.705387 21941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:40.705641 21941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:40.707715 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "aa51c27b935f43e3bdd59497373ff70d"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "c098196810d415d4e99878e5ffeb750e"
server_key_iv: "dd9885abe9ac1599cac683fee9bdd916"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.708310 21941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "aa51c27b935f43e3bdd59497373ff70d"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "c098196810d415d4e99878e5ffeb750e"
server_key_iv: "dd9885abe9ac1599cac683fee9bdd916"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.712064 21941 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:15:40.714558 21955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.715809 21941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:40.715955 21941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "aa51c27b935f43e3bdd59497373ff70d"
format_stamp: "Formatted at 2026-06-20 07:15:40 on dist-test-slave-ldfr"
server_key: "c098196810d415d4e99878e5ffeb750e"
server_key_iv: "dd9885abe9ac1599cac683fee9bdd916"
server_key_version: "encryptionkey@0"
I20260620 07:15:40.716073 21941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:40.734104 21941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:40.734874 21941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:40.735082 21941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:40.735677 21941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:40.736660 21941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:40.736732 21941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.736829 21941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:40.736881 21941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:40.746773 21941 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:44643
I20260620 07:15:40.746793 22068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:44643 every 8 connection(s)
I20260620 07:15:40.747727 21941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:40.751267 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 21941
I20260620 07:15:40.751385 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:40.751662 4052 external_mini_cluster.cc:1428] Setting key eab233423afe3ffec3b252cfd5c15f24
I20260620 07:15:40.757654 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.749546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:54549 (local address 127.3.245.62:36711)
0620 07:15:40.749687 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.749692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.750883 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 07:15:40.752774 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.752777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.752837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.752940 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:40.754039 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.754552 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.755725 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.755935 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.756017 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.757254 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:40.757257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:40.757259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:40.757270 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:40.757288 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:40.757295 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:40.757353 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.757495 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.757545 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:15:40.758519 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.749917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:36711 (local address 127.3.245.3:54549)
0620 07:15:40.750620 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.750662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.752457 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.753166 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.753176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.753278 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:40.753898 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.753920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.754656 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.754658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.755591 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.755600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.756068 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.756070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.756258 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.757007 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:40.757009 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:40.757050 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:40.757112 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:40.757370 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:40.757376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:40.757381 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.757475 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.757700 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":107,"threads_started":1}
I20260620 07:15:40.759629 22069 heartbeater.cc:347] Connected to a master server at 127.3.245.62:36711
I20260620 07:15:40.759888 22069 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:40.760418 22069 heartbeater.cc:511] Master 127.3.245.62:36711 requested a full tablet report, sending...
I20260620 07:15:40.761449 21619 ts_manager.cc:194] Registered new tserver with Master: aa51c27b935f43e3bdd59497373ff70d (127.3.245.3:44643)
I20260620 07:15:40.762050 21619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.3:54549
I20260620 07:15:40.765561 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:40.771353 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.767708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.3.245.62:36711)
0620 07:15:40.767825 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.767829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.768000 (+ 171us) server_negotiation.cc:407] Connection header received
0620 07:15:40.768083 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.768086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.768130 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.768189 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:40.768440 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.769031 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.769857 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.770076 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.770161 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.770903 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:40.770906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:40.770908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:40.770919 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:40.770936 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:40.770944 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:40.771004 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.771173 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.771247 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:15:40.774319 21619 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:40.777339 21619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:40.791512 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.785953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.3.245.2:42405)
0620 07:15:40.786264 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.786283 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.786428 (+ 145us) server_negotiation.cc:407] Connection header received
0620 07:15:40.786688 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.786694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.786914 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.787080 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:40.787389 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.788253 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.790084 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.790842 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.790966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.791163 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.791248 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.791348 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":132,"threads_started":1}
I20260620 07:15:40.791507 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.785838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:42405 (local address 127.0.0.1:57420)
0620 07:15:40.786283 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.786329 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.786476 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.787091 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.787097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.787125 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:40.787248 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.787253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.788340 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.788344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.789964 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.789973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.790964 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.790967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.791070 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.791106 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.791219 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.791341 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260620 07:15:40.792074 22082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.786834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:44643 (local address 127.0.0.1:48708)
0620 07:15:40.787596 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.787610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.787683 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.788335 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.788338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.788355 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:40.788465 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.788470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.789801 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.789805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.790662 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.790671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.791672 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.791676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.791780 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.791811 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.791858 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.791933 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":22,"thread_start_us":79,"threads_started":1}
I20260620 07:15:40.792119 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.786530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34695 (local address 127.0.0.1:45768)
0620 07:15:40.787367 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.787381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.787464 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.788166 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.788169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.788187 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:40.788307 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.788318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.789400 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.789403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.790504 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.790514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.791779 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.791782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.791876 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.791899 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.791957 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.792010 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260620 07:15:40.792500 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.786679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.3.245.1:34695)
0620 07:15:40.787723 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.787737 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.787764 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:15:40.787840 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.787844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.788023 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.788165 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:40.788411 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.789271 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.790642 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.791423 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.791529 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.791726 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.792246 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.792337 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":116,"threads_started":1}
I20260620 07:15:40.792775 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.787229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.3.245.3:44643)
0620 07:15:40.787930 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.787946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.787965 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:15:40.788030 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.788035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.788188 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.788425 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:40.788555 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.789668 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.790817 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.791543 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.792275 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.792441 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.792487 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.792578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":89,"threads_started":1}
I20260620 07:15:40.795100 22003 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d (DEFAULT_TABLE table=test-table [id=8637c31ff95c48b0b5fa5f84b57744c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:40.795251 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d (DEFAULT_TABLE table=test-table [id=8637c31ff95c48b0b5fa5f84b57744c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:40.795251 21741 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d (DEFAULT_TABLE table=test-table [id=8637c31ff95c48b0b5fa5f84b57744c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:40.796121 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd7ffb5a3624c42b352c61ef7abda1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:40.796236 21872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd7ffb5a3624c42b352c61ef7abda1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:40.796236 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd7ffb5a3624c42b352c61ef7abda1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:40.802372 22086 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Bootstrap starting.
I20260620 07:15:40.803792 22088 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: Bootstrap starting.
I20260620 07:15:40.804919 22087 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Bootstrap starting.
I20260620 07:15:40.805060 22086 tablet_bootstrap.cc:654] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:40.805886 22088 tablet_bootstrap.cc:654] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:40.805961 22086 log.cc:824] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:40.806638 22088 log.cc:824] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:40.806730 22087 tablet_bootstrap.cc:654] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:40.807483 22087 log.cc:824] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:40.808126 22086 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: No bootstrap required, opened a new log
I20260620 07:15:40.808351 22086 ts_tablet_manager.cc:1397] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:15:40.808750 22088 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: No bootstrap required, opened a new log
I20260620 07:15:40.808950 22088 ts_tablet_manager.cc:1397] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 07:15:40.809969 22087 tablet_bootstrap.cc:492] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: No bootstrap required, opened a new log
I20260620 07:15:40.810148 22087 ts_tablet_manager.cc:1397] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:15:40.812142 22088 raft_consensus.cc:348] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.812395 22088 raft_consensus.cc:374] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:40.812448 22088 raft_consensus.cc:729] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de151ad246c94edb9ee8a8796d0867c3, State: Initialized, Role: FOLLOWER
I20260620 07:15:40.812783 22086 raft_consensus.cc:348] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.812916 22088 consensus_queue.cc:260] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.813024 22086 raft_consensus.cc:374] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:40.813086 22086 raft_consensus.cc:729] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa51c27b935f43e3bdd59497373ff70d, State: Initialized, Role: FOLLOWER
I20260620 07:15:40.813486 22086 consensus_queue.cc:260] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.813812 21938 heartbeater.cc:503] Master 127.3.245.62:36711 was elected leader, sending a full tablet report...
I20260620 07:15:40.813882 22087 raft_consensus.cc:348] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.814100 22087 raft_consensus.cc:374] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:40.814179 22087 raft_consensus.cc:729] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9439e6726d4443855c20513ab3a5b3, State: Initialized, Role: FOLLOWER
I20260620 07:15:40.814550 22086 ts_tablet_manager.cc:1428] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:15:40.814550 22087 consensus_queue.cc:260] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.814888 22069 heartbeater.cc:503] Master 127.3.245.62:36711 was elected leader, sending a full tablet report...
I20260620 07:15:40.815405 21807 heartbeater.cc:503] Master 127.3.245.62:36711 was elected leader, sending a full tablet report...
I20260620 07:15:40.815451 22087 ts_tablet_manager.cc:1428] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 07:15:40.815980 22088 ts_tablet_manager.cc:1428] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:15:40.824384 22093 raft_consensus.cc:482] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:40.824546 22093 raft_consensus.cc:504] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.825717 22093 leader_election.cc:288] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de151ad246c94edb9ee8a8796d0867c3 (127.3.245.2:42405), ae9439e6726d4443855c20513ab3a5b3 (127.3.245.1:34695)
I20260620 07:15:40.829432 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.826053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:38915 (local address 127.3.245.2:42405)
0620 07:15:40.826219 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.826224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.826237 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:40.826316 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.826319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.826359 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.826443 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:40.826753 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.827264 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.828365 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.828858 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.828976 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.829037 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.829242 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.829311 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:15:40.829454 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.825888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:42405 (local address 127.3.245.3:38915)
0620 07:15:40.826019 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.826037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.826132 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.826468 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.826471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.826488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:40.826619 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.826627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.827381 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.827385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.828223 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.828232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.828981 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.828984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.829120 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.829139 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.829228 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.829298 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:15:40.829885 22095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.826167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34695 (local address 127.3.245.3:49249)
0620 07:15:40.826555 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.826576 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:40.826651 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:40.826971 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:40.826973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:40.826988 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:40.827102 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.827107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.827799 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.827802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.828694 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:40.828704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.829373 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.829376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:40.829535 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.829570 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:15:40.829678 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 07:15:40.829741 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":190,"threads_started":1}
I20260620 07:15:40.829944 22083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.826629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:49249 (local address 127.3.245.1:34695)
0620 07:15:40.826772 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.826776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.826789 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:40.826837 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.826839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.826875 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.826950 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:40.827207 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.827688 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.828811 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.829245 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.829365 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.829445 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.829725 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.829810 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:15:40.830235 21892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd7ffb5a3624c42b352c61ef7abda1d" candidate_uuid: "aa51c27b935f43e3bdd59497373ff70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de151ad246c94edb9ee8a8796d0867c3" is_pre_election: true
I20260620 07:15:40.830513 21761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd7ffb5a3624c42b352c61ef7abda1d" candidate_uuid: "aa51c27b935f43e3bdd59497373ff70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9439e6726d4443855c20513ab3a5b3" is_pre_election: true
I20260620 07:15:40.830544 21892 raft_consensus.cc:2454] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa51c27b935f43e3bdd59497373ff70d in term 0.
I20260620 07:15:40.830785 21761 raft_consensus.cc:2454] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa51c27b935f43e3bdd59497373ff70d in term 0.
I20260620 07:15:40.831071 21958 leader_election.cc:302] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa51c27b935f43e3bdd59497373ff70d, de151ad246c94edb9ee8a8796d0867c3; no voters:
I20260620 07:15:40.831355 22093 raft_consensus.cc:2781] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:40.831429 22093 raft_consensus.cc:482] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:40.831458 22093 raft_consensus.cc:3037] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:40.832741 22093 raft_consensus.cc:504] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.833101 22093 leader_election.cc:288] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [CANDIDATE]: Term 1 election: Requested vote from peers de151ad246c94edb9ee8a8796d0867c3 (127.3.245.2:42405), ae9439e6726d4443855c20513ab3a5b3 (127.3.245.1:34695)
I20260620 07:15:40.833461 21761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd7ffb5a3624c42b352c61ef7abda1d" candidate_uuid: "aa51c27b935f43e3bdd59497373ff70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9439e6726d4443855c20513ab3a5b3"
I20260620 07:15:40.833474 21892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd7ffb5a3624c42b352c61ef7abda1d" candidate_uuid: "aa51c27b935f43e3bdd59497373ff70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de151ad246c94edb9ee8a8796d0867c3"
I20260620 07:15:40.833595 21761 raft_consensus.cc:3037] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:40.833606 21892 raft_consensus.cc:3037] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:40.834614 21761 raft_consensus.cc:2454] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa51c27b935f43e3bdd59497373ff70d in term 1.
I20260620 07:15:40.834649 21892 raft_consensus.cc:2454] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa51c27b935f43e3bdd59497373ff70d in term 1.
I20260620 07:15:40.834928 21958 leader_election.cc:302] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa51c27b935f43e3bdd59497373ff70d, de151ad246c94edb9ee8a8796d0867c3; no voters:
I20260620 07:15:40.835107 22093 raft_consensus.cc:2781] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:40.835335 22093 raft_consensus.cc:686] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 1 LEADER]: Becoming Leader. State: Replica: aa51c27b935f43e3bdd59497373ff70d, State: Running, Role: LEADER
I20260620 07:15:40.835695 22093 consensus_queue.cc:237] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } }
I20260620 07:15:40.838881 21619 catalog_manager.cc:5495] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d reported cstate change: term changed from 0 to 1, leader changed from <none> to aa51c27b935f43e3bdd59497373ff70d (127.3.245.3). New cstate: current_term: 1 leader_uuid: "aa51c27b935f43e3bdd59497373ff70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa51c27b935f43e3bdd59497373ff70d" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 44643 } health_report { overall_health: HEALTHY } } }
I20260620 07:15:40.854359 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.850623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.3.245.3:44643)
0620 07:15:40.850817 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:40.850821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:40.850834 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:40.850910 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:40.850912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:40.850963 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:40.851055 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:40.851349 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.851881 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.852951 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:40.853184 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:40.853285 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:40.853565 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:40.854011 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:40.854107 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:40.854184 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:15:40.854232 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 07:15:40.860208 21761 raft_consensus.cc:1261] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 1 FOLLOWER]: Refusing update from remote peer aa51c27b935f43e3bdd59497373ff70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:40.860268 21892 raft_consensus.cc:1261] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 1 FOLLOWER]: Refusing update from remote peer aa51c27b935f43e3bdd59497373ff70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:40.860944 22093 consensus_queue.cc:1035] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9439e6726d4443855c20513ab3a5b3" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:40.861141 22096 consensus_queue.cc:1035] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de151ad246c94edb9ee8a8796d0867c3" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:15:40.862239 21808 tablet.cc:2369] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:40.878400 21619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:40.878613 21619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:40.881254 21619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de596df6eac04ba090c19761a6ea0c32: Sending DeleteTablet for 3 replicas of tablet 6fd7ffb5a3624c42b352c61ef7abda1d
I20260620 07:15:40.882231 21741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:40 UTC) from {username='slave'} at 127.0.0.1:45768
I20260620 07:15:40.882339 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:40 UTC) from {username='slave'} at 127.0.0.1:57420
I20260620 07:15:40.882467 22003 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd7ffb5a3624c42b352c61ef7abda1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:40 UTC) from {username='slave'} at 127.0.0.1:48708
I20260620 07:15:40.882721 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21679
I20260620 07:15:40.882755 22110 tablet_replica.cc:330] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3: stopping tablet replica
I20260620 07:15:40.883095 22110 raft_consensus.cc:2229] T 6fd7ffb5a3624c42b352c61ef7abda1d P ae9439e6726d4443855c20513ab3a5b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:40.883679 22112 tablet_replica.cc:330] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: stopping tablet replica
I20260620 07:15:40.883961 22112 raft_consensus.cc:2229] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:40.884312 22112 raft_consensus.cc:2258] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:40.886425 22112 ts_tablet_manager.cc:1905] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:40.888881 22111 tablet_replica.cc:330] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3: stopping tablet replica
I20260620 07:15:40.889314 22111 raft_consensus.cc:2229] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:40.889735 22111 raft_consensus.cc:2258] T 6fd7ffb5a3624c42b352c61ef7abda1d P de151ad246c94edb9ee8a8796d0867c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:40.890057 22112 ts_tablet_manager.cc:1918] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:40.890167 22112 log.cc:1196] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/6fd7ffb5a3624c42b352c61ef7abda1d
I20260620 07:15:40.890498 22112 ts_tablet_manager.cc:1939] T 6fd7ffb5a3624c42b352c61ef7abda1d P aa51c27b935f43e3bdd59497373ff70d: Deleting consensus metadata
W20260620 07:15:40.891224 21605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:34695 (error 108)
I20260620 07:15:40.891980 21605 catalog_manager.cc:4841] TS aa51c27b935f43e3bdd59497373ff70d (127.3.245.3:44643): tablet 6fd7ffb5a3624c42b352c61ef7abda1d (table test-table [id=8637c31ff95c48b0b5fa5f84b57744c9]) successfully deleted
I20260620 07:15:40.892056 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21810
I20260620 07:15:40.893887 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.893480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34695 (local address 127.0.0.1:45782)
0620 07:15:40.893696 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.893780 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 07:15:40.894061 21605 catalog_manager.cc:4584] TS ae9439e6726d4443855c20513ab3a5b3 (127.3.245.1:34695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fd7ffb5a3624c42b352c61ef7abda1d: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34695: connect: Connection refused (error 111)
I20260620 07:15:40.899247 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21941
I20260620 07:15:40.901499 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:40.901140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:42405 (local address 127.0.0.1:57424)
0620 07:15:40.901303 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:40.901380 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 07:15:40.906102 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 21588
2026-06-20T07:15:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1025 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:40 dist-test-slave-ldfr krb5kdc[22116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:40 dist-test-slave-ldfr krb5kdc[22116](info): set up 2 sockets
Jun 20 07:15:40 dist-test-slave-ldfr krb5kdc[22116](info): commencing operation
krb5kdc: starting...
W20260620 07:15:41.956768 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:15:42.079973 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44103
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:44103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:42.197180 22129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:42.197520 22129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:42.197604 22129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:42.201707 22129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:42.201810 22129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:42.201851 22129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:42.201905 22129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:42.201941 22129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:42.207494 22129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:44103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:44103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:42.208604 22129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:42.209689 22129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:42.218312 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.218324 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.218291 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:42.218523 22129 server_base.cc:1034] running on GCE node
I20260620 07:15:42.219141 22129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:42.220301 22129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:42.221473 22129 hybrid_clock.cc:648] HybridClock initialized: now 1781939742221430 us; error 53 us; skew 500 ppm
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:42.224344 22129 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:15:42.225317 22129 webserver.cc:460] Webserver started at http://127.3.245.62:37223/ using document root <none> and password file <none>
I20260620 07:15:42.225946 22129 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:42.226042 22129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:42.226272 22129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:42.228368 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "ad99c2376eb6484682a3360e47d2ebcc"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "d7af63d084e3a42ca89a2a47d825c7fd"
server_key_iv: "bb9883dee9de146e47c5e07f2702bc9d"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.228965 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "ad99c2376eb6484682a3360e47d2ebcc"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "d7af63d084e3a42ca89a2a47d825c7fd"
server_key_iv: "bb9883dee9de146e47c5e07f2702bc9d"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.232579 22129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:15:42.235121 22144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.236408 22129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:42.236565 22129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "ad99c2376eb6484682a3360e47d2ebcc"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "d7af63d084e3a42ca89a2a47d825c7fd"
server_key_iv: "bb9883dee9de146e47c5e07f2702bc9d"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.236763 22129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:42.249536 22129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:42.253033 22129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:42.253252 22129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:42.260859 22129 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:44103
I20260620 07:15:42.261044 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:44103 every 8 connection(s)
I20260620 07:15:42.261929 22129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:42.265075 22197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:42.266821 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22129
I20260620 07:15:42.266971 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:42.267308 4052 external_mini_cluster.cc:1428] Setting key fd8549faaec98e0682b0006df20fedd7
I20260620 07:15:42.272029 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: Bootstrap starting.
I20260620 07:15:42.274514 22197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:42.275457 22197 log.cc:824] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:42.277868 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: No bootstrap required, opened a new log
I20260620 07:15:42.281734 22197 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } }
I20260620 07:15:42.281961 22197 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:42.281996 22197 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad99c2376eb6484682a3360e47d2ebcc, State: Initialized, Role: FOLLOWER
I20260620 07:15:42.282215 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:42.269100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.3.245.62:44103)
0620 07:15:42.269617 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:42.269632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:42.269669 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:15:42.270308 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:42.270324 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:42.270650 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:42.271003 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:42.271834 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.272906 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.273768 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.274240 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.274422 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:42.277459 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:42.277464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:42.277478 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:42.277506 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:42.279238 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.279713 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.279724 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.279729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.279838 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.280124 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.280129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.280130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.280533 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:42.280785 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:42.281094 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 07:15:42.281386 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":158,"threads_started":1}
I20260620 07:15:42.282513 22197 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } }
I20260620 07:15:42.282673 22197 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:42.282766 22197 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:42.282866 22197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:42.284093 22197 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } }
I20260620 07:15:42.284456 22197 leader_election.cc:302] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad99c2376eb6484682a3360e47d2ebcc; no voters:
I20260620 07:15:42.284866 22197 leader_election.cc:288] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:42.284950 22202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:42.285245 22202 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [term 1 LEADER]: Becoming Leader. State: Replica: ad99c2376eb6484682a3360e47d2ebcc, State: Running, Role: LEADER
I20260620 07:15:42.285573 22202 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } }
I20260620 07:15:42.286187 22197 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:42.287321 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad99c2376eb6484682a3360e47d2ebcc. Latest consensus state: current_term: 1 leader_uuid: "ad99c2376eb6484682a3360e47d2ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } } }
I20260620 07:15:42.287467 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:42.287724 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad99c2376eb6484682a3360e47d2ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad99c2376eb6484682a3360e47d2ebcc" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 44103 } } }
I20260620 07:15:42.287817 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:42.287976 22211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:42.291196 22211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:42.296933 22211 catalog_manager.cc:1347] Generated new cluster ID: 77c461b7c60f496dab864744714041a8
I20260620 07:15:42.297022 22211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:42.322451 22211 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:42.323536 22211 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:42.329639 22211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: Generated new TSK 0
I20260620 07:15:42.330458 22211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:15:42.363631 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:42.476320 22223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:42.476598 22223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:42.476665 22223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:42.480854 22223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:42.480922 22223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:42.481016 22223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:42.486590 22223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:42.487802 22223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:42.488884 22223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:42.498660 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.498646 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.498649 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:42.498826 22223 server_base.cc:1034] running on GCE node
I20260620 07:15:42.499382 22223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:42.500105 22223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:42.501312 22223 hybrid_clock.cc:648] HybridClock initialized: now 1781939742501282 us; error 63 us; skew 500 ppm
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:42.510509 22223 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:15:42.511480 22223 webserver.cc:460] Webserver started at http://127.3.245.1:45607/ using document root <none> and password file <none>
I20260620 07:15:42.512065 22223 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:42.512168 22223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:42.512404 22223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:42.514542 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "e2b74ab708504c1780ddab878233d1ca"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "4ea9a2e05bc9ae5f0b44e85c19f5f75f"
server_key_iv: "0bcf6cd0f49438879fdd73606c319436"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.515089 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "e2b74ab708504c1780ddab878233d1ca"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "4ea9a2e05bc9ae5f0b44e85c19f5f75f"
server_key_iv: "0bcf6cd0f49438879fdd73606c319436"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.518960 22223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:15:42.521893 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.523283 22223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:42.523443 22223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "e2b74ab708504c1780ddab878233d1ca"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "4ea9a2e05bc9ae5f0b44e85c19f5f75f"
server_key_iv: "0bcf6cd0f49438879fdd73606c319436"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.523576 22223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:42.540284 22223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:42.543959 22223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:42.544180 22223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:42.544817 22223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:42.545697 22223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:42.545766 22223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.545842 22223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:42.545892 22223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.554631 22223 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:34297
I20260620 07:15:42.554729 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:34297 every 8 connection(s)
I20260620 07:15:42.555574 22223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:42.560797 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22223
I20260620 07:15:42.560964 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:42.561295 4052 external_mini_cluster.cc:1428] Setting key 648388ca71e38475216ec27633dfdd75
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:15:42.570607 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:42.557687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:51745 (local address 127.3.245.62:44103)
0620 07:15:42.557860 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:42.557866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:42.558841 (+ 975us) server_negotiation.cc:407] Connection header received
0620 07:15:42.560125 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:42.560140 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:42.560191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:42.560275 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:42.561802 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.562518 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.563779 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.564012 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.564128 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:42.566742 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:42.566745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:42.566747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:42.566773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:42.568340 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.569012 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.569016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.569018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.569104 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.569460 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.569463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.569465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.569657 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:42.569764 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:42.570296 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 07:15:42.570430 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:15:42.571450 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:42.558058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44103 (local address 127.3.245.1:51745)
0620 07:15:42.558662 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:42.558707 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:42.559899 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:42.560477 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:42.560484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:42.561044 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:42.561624 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:42.561647 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.562678 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.562681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:42.563629 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:42.563639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.564176 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.564180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:42.564356 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:42.565048 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:42.565065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:42.566541 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:42.568477 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:42.568484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:42.568501 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:42.568899 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:42.569198 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:42.569202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:42.569204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:42.569355 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:42.569764 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:42.569782 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:42.570045 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 07:15:42.570233 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 07:15:42.570576 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260620 07:15:42.572845 22352 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:15:42.573166 22352 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:42.573812 22352 heartbeater.cc:511] Master 127.3.245.62:44103 requested a full tablet report, sending...
I20260620 07:15:42.575464 22161 ts_manager.cc:194] Registered new tserver with Master: e2b74ab708504c1780ddab878233d1ca (127.3.245.1:34297)
I20260620 07:15:42.576656 22161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:51745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:15:42.591449 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:42.704540 22357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:42.704841 22357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:42.704936 22357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:42.709110 22357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:42.709173 22357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:42.709270 22357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:42.714633 22357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:42.715785 22357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:42.716866 22357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:42.726665 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.726715 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:42.726717 22357 server_base.cc:1034] running on GCE node
W20260620 07:15:42.727013 22365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:42.727455 22357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:42.728148 22357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:42.729341 22357 hybrid_clock.cc:648] HybridClock initialized: now 1781939742729328 us; error 22 us; skew 500 ppm
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:42.732317 22357 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:15:42.733263 22357 webserver.cc:460] Webserver started at http://127.3.245.2:36901/ using document root <none> and password file <none>
I20260620 07:15:42.733841 22357 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:42.733932 22357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:42.734164 22357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:42.736181 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "e37e8b1ecf68f3cc26dc9a5cc69047f4"
server_key_iv: "4d31408009f57fa94f7af05aff15c12f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.736804 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "e37e8b1ecf68f3cc26dc9a5cc69047f4"
server_key_iv: "4d31408009f57fa94f7af05aff15c12f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.740568 22357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:42.743147 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.744585 22357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:42.744767 22357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "e37e8b1ecf68f3cc26dc9a5cc69047f4"
server_key_iv: "4d31408009f57fa94f7af05aff15c12f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.744895 22357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:42.755131 22357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:42.758616 22357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:42.758832 22357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:42.759413 22357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:42.760239 22357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:42.760288 22357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.760354 22357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:42.760386 22357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.770469 22357 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:35571
I20260620 07:15:42.770484 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:35571 every 8 connection(s)
I20260620 07:15:42.771420 22357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:42.778308 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22357
I20260620 07:15:42.778468 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:42.778765 4052 external_mini_cluster.cc:1428] Setting key c954a134e542d9e60cf6b076ecba6dde
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:42.786511 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:42.773499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:37253 (local address 127.3.245.62:44103)
0620 07:15:42.773657 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:42.773660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:42.774753 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 07:15:42.775750 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:42.775754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:42.775815 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:42.775921 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:42.777213 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.777748 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.779068 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.779297 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.779412 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:42.782318 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:42.782325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:42.782330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:42.782368 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:42.783953 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.784683 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.784686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.784688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.784767 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:42.785242 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:42.785249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:42.785254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:42.785553 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:42.785678 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:42.786188 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 07:15:42.786346 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:15:42.787706 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:42.773859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44103 (local address 127.3.245.2:37253)
0620 07:15:42.774579 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:42.774624 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:42.775475 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:42.776105 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:42.776117 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:42.776488 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:42.777053 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:42.777068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.777880 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.777888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:42.778907 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:42.778956 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:42.779458 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:42.779462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:42.779637 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:42.780321 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:42.780338 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:42.782080 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:42.784085 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:42.784094 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:42.784124 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:42.784525 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:42.784923 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:42.784928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:42.784930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:42.785093 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:42.785667 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:42.785673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:42.785930 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 07:15:42.786143 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 07:15:42.786529 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
I20260620 07:15:42.789018 22486 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44103
I20260620 07:15:42.789244 22486 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:42.789710 22486 heartbeater.cc:511] Master 127.3.245.62:44103 requested a full tablet report, sending...
I20260620 07:15:42.790858 22161 ts_manager.cc:194] Registered new tserver with Master: 3af48f47f0f04f4c9236fcfbb0ec1b1d (127.3.245.2:35571)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:15:42.791478 22161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:37253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:15:42.807165 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:42.927227 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:42.927546 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:42.927594 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:42.932252 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:42.932330 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:42.932392 22491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:42.937888 22491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:44103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:42.938970 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:42.939947 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:42.950176 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.950176 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:42.950179 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:42.950444 22491 server_base.cc:1034] running on GCE node
I20260620 07:15:42.950945 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:42.951593 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:42.952767 22491 hybrid_clock.cc:648] HybridClock initialized: now 1781939742952748 us; error 29 us; skew 500 ppm
Jun 20 07:15:42 dist-test-slave-ldfr krb5kdc[22116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:42.955644 22491 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:15:42.956564 22491 webserver.cc:460] Webserver started at http://127.3.245.3:41163/ using document root <none> and password file <none>
I20260620 07:15:42.957167 22491 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:42.957262 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:42.957504 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:42.959560 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "9530f0555a5a4026afdc70b713b41552"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "b4e3ab446611a72c7260d04120747576"
server_key_iv: "5ae14f743edcaf71357394af5de1f88f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.960129 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "9530f0555a5a4026afdc70b713b41552"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "b4e3ab446611a72c7260d04120747576"
server_key_iv: "5ae14f743edcaf71357394af5de1f88f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.964035 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:42.966620 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.968063 22491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:42.968253 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "9530f0555a5a4026afdc70b713b41552"
format_stamp: "Formatted at 2026-06-20 07:15:42 on dist-test-slave-ldfr"
server_key: "b4e3ab446611a72c7260d04120747576"
server_key_iv: "5ae14f743edcaf71357394af5de1f88f"
server_key_version: "encryptionkey@0"
I20260620 07:15:42.968405 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:42.983160 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:42.986928 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:42.987170 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:42.987829 22491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:42.988858 22491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:42.988935 22491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.989008 22491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:42.989068 22491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:42.999375 22491 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:45873
I20260620 07:15:42.999397 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:45873 every 8 connection(s)
I20260620 07:15:43.000398 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:43.003926 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22491
I20260620 07:15:43.004037 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:43.004331 4052 external_mini_cluster.cc:1428] Setting key 9ec9816e4c3b8d06584afa6b0a5e5f5c
Jun 20 07:15:43 dist-test-slave-ldfr krb5kdc[22116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939742, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:43.015784 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.002455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:38729 (local address 127.3.245.62:44103)
0620 07:15:43.002628 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.002632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.003730 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 07:15:43.004884 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.004887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.004943 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.005042 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:43.006935 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.007534 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.008790 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.009013 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.009106 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.011791 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:43.011795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:43.011797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:43.011824 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:43.013453 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:43.014025 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:43.014028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:43.014030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:43.014132 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:43.014599 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:43.014603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:43.014605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:43.014830 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:43.014928 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.015512 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.015649 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:15:43.016968 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.002834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:44103 (local address 127.3.245.3:38729)
0620 07:15:43.003568 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.003612 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.004570 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.005463 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.005470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.006071 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:43.006619 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.006727 (+ 108us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.007681 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.007685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.008667 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.008678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.009179 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.009182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.009359 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.009965 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:43.009982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:43.011614 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:43.013579 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:43.013585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:43.013599 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:43.013909 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:43.014305 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:43.014308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:43.014310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:43.014460 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:43.014967 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:43.014973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:43.015252 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.015428 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.015983 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":118,"threads_started":1}
I20260620 07:15:43.018381 22620 heartbeater.cc:347] Connected to a master server at 127.3.245.62:44103
I20260620 07:15:43.018733 22620 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:43.019336 22620 heartbeater.cc:511] Master 127.3.245.62:44103 requested a full tablet report, sending...
I20260620 07:15:43.020452 22160 ts_manager.cc:194] Registered new tserver with Master: 9530f0555a5a4026afdc70b713b41552 (127.3.245.3:45873)
I20260620 07:15:43.021102 22160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:38729
I20260620 07:15:43.031139 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:43.041699 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.033325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.3.245.62:44103)
0620 07:15:43.033497 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.033501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.033684 (+ 183us) server_negotiation.cc:407] Connection header received
0620 07:15:43.033729 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.033731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.033791 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.033875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:43.034595 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.035131 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.036037 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.036265 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.036356 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.038153 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:43.038156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:43.038159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:43.038184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:43.039627 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:43.040194 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:43.040197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:43.040198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:43.040279 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:43.040824 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:43.040827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:43.040828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:43.041056 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:43.041177 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.041491 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.041575 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:15:43.045308 22160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:43.048437 22160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:43.064030 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.058347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.3.245.2:35571)
0620 07:15:43.058903 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.058915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.058936 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:15:43.059004 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.059009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.059184 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.059584 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:43.059775 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.061153 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.062744 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.063335 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.063463 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.063625 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.063754 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.063860 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":100,"threads_started":1}
I20260620 07:15:43.064146 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.058274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34297 (local address 127.0.0.1:60334)
0620 07:15:43.058883 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.058898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.058980 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.059750 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.059753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.059771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:43.059871 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.059876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.060955 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.060959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.062417 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.062429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.063662 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.063667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.063771 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.063813 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.063869 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.063948 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":69,"thread_start_us":91,"threads_started":1}
I20260620 07:15:43.064025 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.057669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:35571 (local address 127.0.0.1:53916)
0620 07:15:43.058522 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.058539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.058640 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.059365 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.059372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.059401 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:43.059608 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.059619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.061293 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.061296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.062623 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.062631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.063464 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.063467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.063573 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.063609 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.063725 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.063860 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":164,"thread_start_us":117,"threads_started":1}
I20260620 07:15:43.064601 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.058438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.3.245.1:34297)
0620 07:15:43.059218 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.059234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.059259 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:15:43.059336 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.059342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.059594 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.059786 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:43.059994 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.060823 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.062556 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.063503 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.064106 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.064277 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.064331 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.064428 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":86,"threads_started":1}
I20260620 07:15:43.064486 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.057486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45873 (local address 127.0.0.1:49006)
0620 07:15:43.057917 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.057981 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.058108 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.059417 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.059420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.059434 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:43.059593 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.059601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.061336 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.061341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.062665 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.062676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.064081 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.064085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.064173 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.064196 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.064272 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.064346 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":99,"thread_start_us":85,"threads_started":1}
I20260620 07:15:43.064944 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.057614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.3.245.3:45873)
0620 07:15:43.058669 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.058684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.058714 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:15:43.058804 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.058810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.059043 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.059226 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:43.059747 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.061131 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.062829 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.063754 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.063873 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.064015 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.064592 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.064763 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":218,"threads_started":1}
I20260620 07:15:43.067651 22554 tablet_service.cc:1467] Processing CreateTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 (DEFAULT_TABLE table=test-table [id=2025d80d207c4988b4dde8189de2dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:43.067654 22420 tablet_service.cc:1467] Processing CreateTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 (DEFAULT_TABLE table=test-table [id=2025d80d207c4988b4dde8189de2dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:43.067654 22286 tablet_service.cc:1467] Processing CreateTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 (DEFAULT_TABLE table=test-table [id=2025d80d207c4988b4dde8189de2dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:43.068831 22420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580ad7e87fd443ffb772288ab6f2d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:43.068831 22286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580ad7e87fd443ffb772288ab6f2d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:43.068845 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 580ad7e87fd443ffb772288ab6f2d183. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:43.076395 22637 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Bootstrap starting.
I20260620 07:15:43.077656 22638 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Bootstrap starting.
I20260620 07:15:43.078855 22637 tablet_bootstrap.cc:654] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:43.079444 22638 tablet_bootstrap.cc:654] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:43.080081 22637 log.cc:824] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:43.080188 22638 log.cc:824] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:43.080964 22639 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Bootstrap starting.
I20260620 07:15:43.082244 22638 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: No bootstrap required, opened a new log
I20260620 07:15:43.082407 22637 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: No bootstrap required, opened a new log
I20260620 07:15:43.082502 22638 ts_tablet_manager.cc:1397] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:15:43.082624 22637 ts_tablet_manager.cc:1397] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 07:15:43.083664 22639 tablet_bootstrap.cc:654] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:43.084517 22639 log.cc:824] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:43.086611 22637 raft_consensus.cc:348] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.086900 22637 raft_consensus.cc:374] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:43.086928 22639 tablet_bootstrap.cc:492] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: No bootstrap required, opened a new log
I20260620 07:15:43.086999 22637 raft_consensus.cc:729] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9530f0555a5a4026afdc70b713b41552, State: Initialized, Role: FOLLOWER
I20260620 07:15:43.087173 22639 ts_tablet_manager.cc:1397] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 07:15:43.086941 22638 raft_consensus.cc:348] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.087262 22638 raft_consensus.cc:374] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:43.087347 22638 raft_consensus.cc:729] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b74ab708504c1780ddab878233d1ca, State: Initialized, Role: FOLLOWER
I20260620 07:15:43.087548 22637 consensus_queue.cc:260] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.087848 22638 consensus_queue.cc:260] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.088586 22637 ts_tablet_manager.cc:1428] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 07:15:43.088941 22638 ts_tablet_manager.cc:1428] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:15:43.089030 22620 heartbeater.cc:503] Master 127.3.245.62:44103 was elected leader, sending a full tablet report...
I20260620 07:15:43.089298 22352 heartbeater.cc:503] Master 127.3.245.62:44103 was elected leader, sending a full tablet report...
I20260620 07:15:43.091142 22639 raft_consensus.cc:348] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.091449 22639 raft_consensus.cc:374] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:43.091511 22639 raft_consensus.cc:729] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af48f47f0f04f4c9236fcfbb0ec1b1d, State: Initialized, Role: FOLLOWER
I20260620 07:15:43.092036 22639 consensus_queue.cc:260] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.092803 22486 heartbeater.cc:503] Master 127.3.245.62:44103 was elected leader, sending a full tablet report...
I20260620 07:15:43.093200 22639 ts_tablet_manager.cc:1428] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260620 07:15:43.251487 22621 tablet.cc:2369] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:43.272593 22487 tablet.cc:2369] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:43.307125 22353 tablet.cc:2369] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:43.488477 22644 raft_consensus.cc:482] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:43.488765 22644 raft_consensus.cc:504] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.489955 22644 leader_election.cc:288] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9530f0555a5a4026afdc70b713b41552 (127.3.245.3:45873), 3af48f47f0f04f4c9236fcfbb0ec1b1d (127.3.245.2:35571)
I20260620 07:15:43.495534 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.490413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:32949 (local address 127.3.245.2:35571)
0620 07:15:43.490551 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.490556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.491090 (+ 534us) server_negotiation.cc:407] Connection header received
0620 07:15:43.491257 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.491263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.491310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.491375 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:43.491661 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.492391 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494085 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.494702 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494822 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.494890 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.495290 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.495378 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:15:43.495534 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.490428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:51859 (local address 127.3.245.3:45873)
0620 07:15:43.490560 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.490567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.490807 (+ 240us) server_negotiation.cc:407] Connection header received
0620 07:15:43.490987 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.490990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.491070 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.491155 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:43.491506 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.492283 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494085 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.494702 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494822 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.494891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.495290 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.495378 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:43.495643 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.490304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:35571 (local address 127.3.245.1:32949)
0620 07:15:43.490970 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.490983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.491106 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.491427 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.491433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.491455 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:43.491556 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.491561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.492491 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.492494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.493924 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.493937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.494870 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.495064 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.495142 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.495230 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.495426 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":234,"thread_start_us":107,"threads_started":1}
I20260620 07:15:43.495643 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.490304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:45873 (local address 127.3.245.1:51859)
0620 07:15:43.490698 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.490712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:43.490864 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:43.491208 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:43.491211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:43.491231 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:43.491377 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.491387 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.492408 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.492411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.493924 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:43.493937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.494870 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.494874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:43.495087 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.495147 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:15:43.495230 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 07:15:43.495429 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":211,"thread_start_us":118,"threads_started":1}
I20260620 07:15:43.496507 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580ad7e87fd443ffb772288ab6f2d183" candidate_uuid: "e2b74ab708504c1780ddab878233d1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" is_pre_election: true
I20260620 07:15:43.496773 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580ad7e87fd443ffb772288ab6f2d183" candidate_uuid: "e2b74ab708504c1780ddab878233d1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9530f0555a5a4026afdc70b713b41552" is_pre_election: true
I20260620 07:15:43.497054 22440 raft_consensus.cc:2454] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b74ab708504c1780ddab878233d1ca in term 0.
I20260620 07:15:43.497063 22574 raft_consensus.cc:2454] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b74ab708504c1780ddab878233d1ca in term 0.
I20260620 07:15:43.497618 22240 leader_election.cc:302] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af48f47f0f04f4c9236fcfbb0ec1b1d, e2b74ab708504c1780ddab878233d1ca; no voters:
I20260620 07:15:43.498023 22644 raft_consensus.cc:2781] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:43.498114 22644 raft_consensus.cc:482] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:43.498158 22644 raft_consensus.cc:3037] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:43.499338 22644 raft_consensus.cc:504] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.499684 22644 leader_election.cc:288] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [CANDIDATE]: Term 1 election: Requested vote from peers 9530f0555a5a4026afdc70b713b41552 (127.3.245.3:45873), 3af48f47f0f04f4c9236fcfbb0ec1b1d (127.3.245.2:35571)
I20260620 07:15:43.500099 22440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580ad7e87fd443ffb772288ab6f2d183" candidate_uuid: "e2b74ab708504c1780ddab878233d1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d"
I20260620 07:15:43.500119 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580ad7e87fd443ffb772288ab6f2d183" candidate_uuid: "e2b74ab708504c1780ddab878233d1ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9530f0555a5a4026afdc70b713b41552"
I20260620 07:15:43.500224 22440 raft_consensus.cc:3037] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:43.500250 22574 raft_consensus.cc:3037] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:43.501400 22574 raft_consensus.cc:2454] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b74ab708504c1780ddab878233d1ca in term 1.
I20260620 07:15:43.501425 22440 raft_consensus.cc:2454] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b74ab708504c1780ddab878233d1ca in term 1.
I20260620 07:15:43.501755 22239 leader_election.cc:302] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9530f0555a5a4026afdc70b713b41552, e2b74ab708504c1780ddab878233d1ca; no voters:
I20260620 07:15:43.501986 22644 raft_consensus.cc:2781] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:43.502247 22644 raft_consensus.cc:686] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 1 LEADER]: Becoming Leader. State: Replica: e2b74ab708504c1780ddab878233d1ca, State: Running, Role: LEADER
I20260620 07:15:43.502617 22644 consensus_queue.cc:237] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } }
I20260620 07:15:43.506546 22160 catalog_manager.cc:5495] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b74ab708504c1780ddab878233d1ca (127.3.245.1). New cstate: current_term: 1 leader_uuid: "e2b74ab708504c1780ddab878233d1ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b74ab708504c1780ddab878233d1ca" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 } health_report { overall_health: UNKNOWN } } }
I20260620 07:15:43.519158 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.515309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.3.245.1:34297)
0620 07:15:43.515470 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.515475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.515489 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:15:43.515606 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.515609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.515674 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.515767 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:43.516078 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.516584 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.517648 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.517868 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.517983 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.518308 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:43.518717 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:43.518880 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.518973 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.519012 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:15:43.525236 22574 raft_consensus.cc:1261] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 1 FOLLOWER]: Refusing update from remote peer e2b74ab708504c1780ddab878233d1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:43.525235 22440 raft_consensus.cc:1261] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 1 FOLLOWER]: Refusing update from remote peer e2b74ab708504c1780ddab878233d1ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:43.525940 22648 consensus_queue.cc:1035] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af48f47f0f04f4c9236fcfbb0ec1b1d" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:43.526131 22644 consensus_queue.cc:1035] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "9530f0555a5a4026afdc70b713b41552" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:43.534135 22651 mvcc.cc:205] Tried to move back new op lower bound from 7298825189471539200 to 7298825189389369344. Current Snapshot: MvccSnapshot[applied={T|T < 7298825189471539200}]
I20260620 07:15:43.534747 22652 mvcc.cc:205] Tried to move back new op lower bound from 7298825189471539200 to 7298825189389369344. Current Snapshot: MvccSnapshot[applied={T|T < 7298825189471539200}]
I20260620 07:15:43.543408 22161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:43.543562 22161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:43.546114 22161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad99c2376eb6484682a3360e47d2ebcc: Sending DeleteTablet for 3 replicas of tablet 580ad7e87fd443ffb772288ab6f2d183
I20260620 07:15:43.547008 22286 tablet_service.cc:1514] Processing DeleteTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:43 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:60334
I20260620 07:15:43.547008 22420 tablet_service.cc:1514] Processing DeleteTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:43 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:53916
I20260620 07:15:43.547084 22554 tablet_service.cc:1514] Processing DeleteTablet for tablet 580ad7e87fd443ffb772288ab6f2d183 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:43 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:49006
I20260620 07:15:43.547559 22662 tablet_replica.cc:330] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: stopping tablet replica
I20260620 07:15:43.547943 22660 tablet_replica.cc:330] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca: stopping tablet replica
I20260620 07:15:43.548017 22662 raft_consensus.cc:2229] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:43.548264 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22223
I20260620 07:15:43.548269 22660 raft_consensus.cc:2229] T 580ad7e87fd443ffb772288ab6f2d183 P e2b74ab708504c1780ddab878233d1ca [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:43.549762 22662 raft_consensus.cc:2258] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:43.550205 22661 tablet_replica.cc:330] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: stopping tablet replica
I20260620 07:15:43.550529 22661 raft_consensus.cc:2229] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:43.550855 22661 raft_consensus.cc:2258] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:43.551383 22662 ts_tablet_manager.cc:1905] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:43.552273 22661 ts_tablet_manager.cc:1905] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:43.554296 22662 ts_tablet_manager.cc:1918] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:43.554409 22662 log.cc:1196] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/580ad7e87fd443ffb772288ab6f2d183
I20260620 07:15:43.554762 22662 ts_tablet_manager.cc:1939] T 580ad7e87fd443ffb772288ab6f2d183 P 9530f0555a5a4026afdc70b713b41552: Deleting consensus metadata
I20260620 07:15:43.555213 22661 ts_tablet_manager.cc:1918] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:43.555353 22661 log.cc:1196] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/580ad7e87fd443ffb772288ab6f2d183
W20260620 07:15:43.555312 22145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:34297 (error 108)
I20260620 07:15:43.555675 22661 ts_tablet_manager.cc:1939] T 580ad7e87fd443ffb772288ab6f2d183 P 3af48f47f0f04f4c9236fcfbb0ec1b1d: Deleting consensus metadata
I20260620 07:15:43.556318 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.555997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34297 (local address 127.0.0.1:60344)
0620 07:15:43.556126 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.556203 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:15:43.556445 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22357
W20260620 07:15:43.556504 22145 catalog_manager.cc:4584] TS e2b74ab708504c1780ddab878233d1ca (127.3.245.1:34297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 580ad7e87fd443ffb772288ab6f2d183: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34297: connect: Connection refused (error 111)
I20260620 07:15:43.556723 22145 catalog_manager.cc:4841] TS 9530f0555a5a4026afdc70b713b41552 (127.3.245.3:45873): tablet 580ad7e87fd443ffb772288ab6f2d183 (table test-table [id=2025d80d207c4988b4dde8189de2dbfb]) successfully deleted
I20260620 07:15:43.562253 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22491
I20260620 07:15:43.563419 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.562392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:35571 (local address 127.0.0.1:53922)
0620 07:15:43.563286 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:43.563346 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:35571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":807}
I20260620 07:15:43.568027 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22129
2026-06-20T07:15:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2662 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T07:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:43Z Disabled control of system clock
I20260620 07:15:43.599108 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41131
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:43.710621 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:43.710891 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:43.710925 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:43.714955 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:43.715020 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:43.715035 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:43.715052 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:43.715067 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:43.719822 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:41131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:43.720927 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:43.722283 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:43.730954 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:43.730993 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:43.731009 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:43.731124 22667 server_base.cc:1034] running on GCE node
I20260620 07:15:43.731838 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:43.733058 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:43.734230 22667 hybrid_clock.cc:648] HybridClock initialized: now 1781939743734197 us; error 46 us; skew 500 ppm
I20260620 07:15:43.736264 22667 webserver.cc:460] Webserver started at http://127.3.245.62:37853/ using document root <none> and password file <none>
I20260620 07:15:43.736914 22667 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:43.737008 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:43.737236 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:43.739331 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "5fd5ddb8613b4027b5b9b1e363737822"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "0782ca114f09ef28d1054f5e189f8e91"
server_key_iv: "13a3917fae471bfe38f60c3144ef55ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.739902 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "5fd5ddb8613b4027b5b9b1e363737822"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "0782ca114f09ef28d1054f5e189f8e91"
server_key_iv: "13a3917fae471bfe38f60c3144ef55ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.743698 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 07:15:43.746217 22681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:43.747488 22667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:15:43.747629 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "5fd5ddb8613b4027b5b9b1e363737822"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "0782ca114f09ef28d1054f5e189f8e91"
server_key_iv: "13a3917fae471bfe38f60c3144ef55ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.747746 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:43.768153 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:43.768980 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:43.769204 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:43.776209 22667 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:41131
I20260620 07:15:43.776235 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:41131 every 8 connection(s)
I20260620 07:15:43.777196 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:43.780388 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:43.785310 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22667
I20260620 07:15:43.785472 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:43.785738 4052 external_mini_cluster.cc:1428] Setting key 2da8e03b6523c502fb2f657432b5a4bb
I20260620 07:15:43.786142 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: Bootstrap starting.
I20260620 07:15:43.788719 22734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:43.789472 22734 log.cc:824] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:43.791505 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: No bootstrap required, opened a new log
I20260620 07:15:43.795140 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:43.787231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.3.245.62:41131)
0620 07:15:43.787905 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:43.787920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:43.787956 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:15:43.788600 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:43.788670 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:43.789121 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:43.789457 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:43.789924 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.791009 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.791971 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:43.792442 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:43.792616 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:43.793478 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:43.793486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:43.793505 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:43.793883 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:43.793935 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:43.793954 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:43.794064 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:43.794175 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 07:15:43.794434 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260620 07:15:43.795104 22734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } }
I20260620 07:15:43.795410 22734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:43.795476 22734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd5ddb8613b4027b5b9b1e363737822, State: Initialized, Role: FOLLOWER
I20260620 07:15:43.795921 22734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } }
I20260620 07:15:43.796077 22734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:43.796145 22734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:43.796278 22734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:43.797317 22734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } }
I20260620 07:15:43.797652 22734 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd5ddb8613b4027b5b9b1e363737822; no voters:
I20260620 07:15:43.797996 22734 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:43.798111 22739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:43.798364 22739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd5ddb8613b4027b5b9b1e363737822, State: Running, Role: LEADER
I20260620 07:15:43.798715 22739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } }
I20260620 07:15:43.799237 22734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:43.800366 22741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd5ddb8613b4027b5b9b1e363737822. Latest consensus state: current_term: 1 leader_uuid: "5fd5ddb8613b4027b5b9b1e363737822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } } }
I20260620 07:15:43.800496 22741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:43.800832 22740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd5ddb8613b4027b5b9b1e363737822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd5ddb8613b4027b5b9b1e363737822" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41131 } } }
I20260620 07:15:43.801009 22740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:43.801357 22748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:43.804236 22748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:43.809928 22748 catalog_manager.cc:1347] Generated new cluster ID: 44f010d959d54eb4bfae3a50753e94b0
I20260620 07:15:43.810021 22748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:43.825690 22748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:43.826694 22748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:43.831903 22748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: Generated new TSK 0
I20260620 07:15:43.832798 22748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:43.843214 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--builtin_ntp_servers=127.3.245.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:43.957719 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:43.958071 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:43.958154 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:43.962401 22758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:43.962505 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:43.962600 22758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:43.967664 22758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:43.968914 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:43.970366 22758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:43.980015 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:43.980015 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:43.980039 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:43.980309 22758 server_base.cc:1034] running on GCE node
I20260620 07:15:43.980818 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:43.981436 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:43.982632 22758 hybrid_clock.cc:648] HybridClock initialized: now 1781939743982608 us; error 36 us; skew 500 ppm
I20260620 07:15:43.984550 22758 webserver.cc:460] Webserver started at http://127.3.245.1:35891/ using document root <none> and password file <none>
I20260620 07:15:43.985157 22758 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:43.985250 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:43.985487 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:43.987639 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "5c636e427bf844f2a3b6e5ae3e52c560"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "299804faf29a62a9634aa7f6a3d4ebdf"
server_key_iv: "5bdccec457d1b7ddc5ec39b25cee812a"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.988240 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "5c636e427bf844f2a3b6e5ae3e52c560"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "299804faf29a62a9634aa7f6a3d4ebdf"
server_key_iv: "5bdccec457d1b7ddc5ec39b25cee812a"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.992019 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:43.994793 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:43.996171 22758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:43.996346 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "5c636e427bf844f2a3b6e5ae3e52c560"
format_stamp: "Formatted at 2026-06-20 07:15:43 on dist-test-slave-ldfr"
server_key: "299804faf29a62a9634aa7f6a3d4ebdf"
server_key_iv: "5bdccec457d1b7ddc5ec39b25cee812a"
server_key_version: "encryptionkey@0"
I20260620 07:15:43.996480 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:44.011194 22758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:44.011929 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:44.012137 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:44.012797 22758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:44.013736 22758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:44.013818 22758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.013918 22758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:44.013972 22758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.022876 22758 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:34245
I20260620 07:15:44.022945 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:34245 every 8 connection(s)
I20260620 07:15:44.023835 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:44.030316 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22758
I20260620 07:15:44.030458 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:44.030777 4052 external_mini_cluster.cc:1428] Setting key 03b22ed0d8b0488349608ddc89fec1f5
I20260620 07:15:44.032892 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--builtin_ntp_servers=127.3.245.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:44.034008 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.025795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:43627 (local address 127.3.245.62:41131)
0620 07:15:44.025936 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.025940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.027072 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 07:15:44.028456 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.028458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.028503 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.028578 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:44.029532 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.030106 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.031714 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.032055 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.032167 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.033517 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:44.033520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:44.033522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:44.033533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:44.033551 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:44.033559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:44.033636 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.033809 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.033871 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:44.034916 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.026184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41131 (local address 127.3.245.1:43627)
0620 07:15:44.026906 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.026949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.028256 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.028781 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.028788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.028877 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:44.029403 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.029417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.030280 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.030284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.031596 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.031606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.032238 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.032242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.032469 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.033211 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:44.033213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:44.033252 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:44.033345 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:44.033651 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:44.033661 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:44.033667 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.033807 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.034096 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":113,"threads_started":1}
I20260620 07:15:44.036141 22886 heartbeater.cc:347] Connected to a master server at 127.3.245.62:41131
I20260620 07:15:44.036389 22886 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:44.036962 22886 heartbeater.cc:511] Master 127.3.245.62:41131 requested a full tablet report, sending...
I20260620 07:15:44.038426 22698 ts_manager.cc:194] Registered new tserver with Master: 5c636e427bf844f2a3b6e5ae3e52c560 (127.3.245.1:34245)
I20260620 07:15:44.039553 22698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.1:43627
W20260620 07:15:44.147395 22889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:44.147677 22889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:44.147717 22889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:44.152017 22889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:44.152081 22889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:44.152139 22889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:44.157146 22889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:44.158208 22889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:44.159631 22889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:44.169473 22894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:44.169523 22895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:44.169500 22897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:44.169543 22889 server_base.cc:1034] running on GCE node
I20260620 07:15:44.170166 22889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:44.170898 22889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:44.172107 22889 hybrid_clock.cc:648] HybridClock initialized: now 1781939744172072 us; error 48 us; skew 500 ppm
I20260620 07:15:44.174175 22889 webserver.cc:460] Webserver started at http://127.3.245.2:43561/ using document root <none> and password file <none>
I20260620 07:15:44.174783 22889 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:44.174878 22889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:44.175112 22889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:44.177261 22889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "259a699c8bad4dd695c145c32d5cd775"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "4adf13c61d27df7a1cdd7c48126d0460"
server_key_iv: "e1d4a3253631e0a768f60eeec2546140"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.177825 22889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "259a699c8bad4dd695c145c32d5cd775"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "4adf13c61d27df7a1cdd7c48126d0460"
server_key_iv: "e1d4a3253631e0a768f60eeec2546140"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.181803 22889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:15:44.184381 22903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.185791 22889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:44.185945 22889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "259a699c8bad4dd695c145c32d5cd775"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "4adf13c61d27df7a1cdd7c48126d0460"
server_key_iv: "e1d4a3253631e0a768f60eeec2546140"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.186071 22889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:44.203132 22889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:44.203882 22889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:44.204092 22889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:44.204741 22889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:44.205687 22889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:44.205758 22889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.205825 22889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:44.205868 22889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.214838 22889 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:45263
I20260620 07:15:44.214867 23016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:45263 every 8 connection(s)
I20260620 07:15:44.215765 22889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:44.219666 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 22889
I20260620 07:15:44.219764 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:44.220036 4052 external_mini_cluster.cc:1428] Setting key 60f539ec370df55036f7566238472e4a
I20260620 07:15:44.222101 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--builtin_ntp_servers=127.3.245.20:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:44.225912 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.217662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:47233 (local address 127.3.245.62:41131)
0620 07:15:44.217815 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.217820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.218921 (+ 1101us) server_negotiation.cc:407] Connection header received
0620 07:15:44.220447 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.220450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.220509 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.220601 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:44.221686 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.222485 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.223650 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.223879 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.223991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.225275 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:44.225280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:44.225283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:44.225306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:44.225336 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:44.225345 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:44.225436 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.225659 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.225731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:15:44.226883 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.218132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41131 (local address 127.3.245.2:47233)
0620 07:15:44.218759 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.218807 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.220233 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.220825 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.220831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.220914 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:44.221537 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.221555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.222608 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.222611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.223512 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.223522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.224030 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.224034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.224200 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.224951 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:44.224953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:44.224995 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:44.225084 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:44.225441 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:44.225453 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:44.225462 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.225659 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.225961 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260620 07:15:44.228361 23017 heartbeater.cc:347] Connected to a master server at 127.3.245.62:41131
I20260620 07:15:44.228688 23017 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:44.229234 23017 heartbeater.cc:511] Master 127.3.245.62:41131 requested a full tablet report, sending...
I20260620 07:15:44.230403 22698 ts_manager.cc:194] Registered new tserver with Master: 259a699c8bad4dd695c145c32d5cd775 (127.3.245.2:45263)
I20260620 07:15:44.231300 22698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.2:47233
W20260620 07:15:44.340052 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:44.340314 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:44.340353 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:44.344372 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:44.344437 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:44.344492 23020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:44.349848 23020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:44.351212 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:44.353051 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:44.362777 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:44.362831 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:44.362881 23020 server_base.cc:1034] running on GCE node
W20260620 07:15:44.363099 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:44.363526 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:44.364195 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:44.365387 23020 hybrid_clock.cc:648] HybridClock initialized: now 1781939744365369 us; error 29 us; skew 500 ppm
I20260620 07:15:44.367259 23020 webserver.cc:460] Webserver started at http://127.3.245.3:46369/ using document root <none> and password file <none>
I20260620 07:15:44.367841 23020 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:44.367900 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:44.368160 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:44.370324 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "d3ffde88bd2b4373a7ea59b84c871459"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "491a425854c1959989a140a05695742c"
server_key_iv: "e4f00a167ce10a849775130a626e6e29"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.370877 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "d3ffde88bd2b4373a7ea59b84c871459"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "491a425854c1959989a140a05695742c"
server_key_iv: "e4f00a167ce10a849775130a626e6e29"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.374543 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 07:15:44.377131 23034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.378364 23020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:44.378486 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "d3ffde88bd2b4373a7ea59b84c871459"
format_stamp: "Formatted at 2026-06-20 07:15:44 on dist-test-slave-ldfr"
server_key: "491a425854c1959989a140a05695742c"
server_key_iv: "e4f00a167ce10a849775130a626e6e29"
server_key_version: "encryptionkey@0"
I20260620 07:15:44.378597 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:44.390234 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:44.390889 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:44.391083 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:44.391602 23020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:44.392477 23020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:44.392522 23020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.392560 23020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:44.392585 23020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:44.401407 23020 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:40425
I20260620 07:15:44.401486 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:40425 every 8 connection(s)
I20260620 07:15:44.402364 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:44.408917 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23020
I20260620 07:15:44.409011 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:44.409272 4052 external_mini_cluster.cc:1428] Setting key 633068727eebbfb3a38b6a8a7cbf5e06
I20260620 07:15:44.411969 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.404393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:47605 (local address 127.3.245.62:41131)
0620 07:15:44.404558 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.404562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.405673 (+ 1111us) server_negotiation.cc:407] Connection header received
0620 07:15:44.407047 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.407050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.407107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.407213 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:44.408079 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.408604 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.410021 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.410253 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.410363 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.411474 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:44.411478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:44.411481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:44.411493 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:44.411518 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:44.411526 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:44.411600 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.411764 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.411834 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:15:44.412969 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.404844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41131 (local address 127.3.245.3:47605)
0620 07:15:44.405517 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.405556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.406841 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.407386 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.407393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.407473 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:44.407936 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.407948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.408797 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.408802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.409887 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.409898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.410398 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.410402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.410587 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.411209 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:44.411211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:44.411250 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:44.411319 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:44.411613 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:44.411624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:44.411632 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.411761 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.412039 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260620 07:15:44.413975 23148 heartbeater.cc:347] Connected to a master server at 127.3.245.62:41131
I20260620 07:15:44.414208 23148 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:44.414731 23148 heartbeater.cc:511] Master 127.3.245.62:41131 requested a full tablet report, sending...
I20260620 07:15:44.415728 22698 ts_manager.cc:194] Registered new tserver with Master: d3ffde88bd2b4373a7ea59b84c871459 (127.3.245.3:40425)
I20260620 07:15:44.416373 22698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.3:47605
I20260620 07:15:44.423026 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:44.428449 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.424923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.3.245.62:41131)
0620 07:15:44.425054 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.425058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.425239 (+ 181us) server_negotiation.cc:407] Connection header received
0620 07:15:44.425345 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.425347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.425385 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.425445 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:44.425665 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.426180 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.427059 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.427263 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.427344 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.428069 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:44.428072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:44.428074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:44.428085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:44.428102 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:44.428109 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:44.428207 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.428313 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.428344 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:44.431187 22698 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:44.434353 22698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:44.449812 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.443840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.3.245.2:45263)
0620 07:15:44.444348 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.444360 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.444760 (+ 400us) server_negotiation.cc:407] Connection header received
0620 07:15:44.444849 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.444855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.445077 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.445259 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:44.445564 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446751 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.448210 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449018 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.449158 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.449376 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.449515 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.449629 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":114,"threads_started":1}
I20260620 07:15:44.449806 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.443469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:45263 (local address 127.0.0.1:58448)
0620 07:15:44.444451 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.444468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.444589 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.445245 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.445251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.445276 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:44.445426 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.445432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446892 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.446897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.448056 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.448066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449156 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.449160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.449281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.449318 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.449450 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.449617 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":42,"spinlock_wait_cycles":3328,"thread_start_us":123,"threads_started":1}
I20260620 07:15:44.449812 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.443402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40425 (local address 127.0.0.1:39646)
0620 07:15:44.444386 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.444440 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.444589 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.445323 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.445326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.445344 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:44.445458 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.445463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446988 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.446991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.448211 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.448219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449255 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.449259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.449346 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.449369 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.449423 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.449603 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":99,"spinlock_wait_cycles":7680,"thread_start_us":96,"threads_started":1}
I20260620 07:15:44.449813 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.444242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34245 (local address 127.0.0.1:40044)
0620 07:15:44.444922 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.444937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.445031 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.445571 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.445574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.445590 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:44.445664 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.445669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446623 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.446629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.448033 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.448044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449453 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.449458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.449548 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.449568 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.449634 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.449699 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260620 07:15:44.450492 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.444688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.3.245.1:34245)
0620 07:15:44.445104 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.445116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.445140 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:15:44.445244 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.445250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.445436 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.445593 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:44.445756 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446496 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.448192 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449075 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.449201 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.449407 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.450118 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.450238 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":111,"threads_started":1}
I20260620 07:15:44.450877 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.443515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.3.245.3:40425)
0620 07:15:44.444922 (+ 1407us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.444934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.444954 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:15:44.445017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.445022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.445176 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.445640 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:44.445794 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.446876 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.448321 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.449108 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.450296 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.450536 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.450589 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.450683 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":96,"threads_started":1}
I20260620 07:15:44.453365 23082 tablet_service.cc:1467] Processing CreateTablet for tablet 0540047e87884689ab26fa7a5785c16b (DEFAULT_TABLE table=test-table [id=16929f823ad044ed900e3cba4fb47131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:44.453352 22820 tablet_service.cc:1467] Processing CreateTablet for tablet 0540047e87884689ab26fa7a5785c16b (DEFAULT_TABLE table=test-table [id=16929f823ad044ed900e3cba4fb47131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:44.453351 22951 tablet_service.cc:1467] Processing CreateTablet for tablet 0540047e87884689ab26fa7a5785c16b (DEFAULT_TABLE table=test-table [id=16929f823ad044ed900e3cba4fb47131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:44.454360 23082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0540047e87884689ab26fa7a5785c16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:44.454376 22820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0540047e87884689ab26fa7a5785c16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:44.454376 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0540047e87884689ab26fa7a5785c16b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:44.461247 23165 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Bootstrap starting.
I20260620 07:15:44.462497 23166 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Bootstrap starting.
I20260620 07:15:44.463526 23165 tablet_bootstrap.cc:654] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:44.464283 23165 log.cc:824] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:44.464421 23166 tablet_bootstrap.cc:654] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:44.465015 23167 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Bootstrap starting.
I20260620 07:15:44.465324 23166 log.cc:824] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:44.466686 23165 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: No bootstrap required, opened a new log
I20260620 07:15:44.466914 23165 ts_tablet_manager.cc:1397] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:15:44.467381 23167 tablet_bootstrap.cc:654] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:44.467888 23166 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: No bootstrap required, opened a new log
I20260620 07:15:44.468089 23166 ts_tablet_manager.cc:1397] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 07:15:44.468259 23167 log.cc:824] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:44.470369 23167 tablet_bootstrap.cc:492] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: No bootstrap required, opened a new log
I20260620 07:15:44.470554 23167 ts_tablet_manager.cc:1397] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 07:15:44.470794 23165 raft_consensus.cc:348] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.471041 23165 raft_consensus.cc:374] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:44.471093 23165 raft_consensus.cc:729] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ffde88bd2b4373a7ea59b84c871459, State: Initialized, Role: FOLLOWER
I20260620 07:15:44.471545 23165 consensus_queue.cc:260] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.472241 23148 heartbeater.cc:503] Master 127.3.245.62:41131 was elected leader, sending a full tablet report...
I20260620 07:15:44.472560 23165 ts_tablet_manager.cc:1428] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:15:44.471977 23166 raft_consensus.cc:348] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.472900 23166 raft_consensus.cc:374] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:44.472957 23166 raft_consensus.cc:729] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c636e427bf844f2a3b6e5ae3e52c560, State: Initialized, Role: FOLLOWER
I20260620 07:15:44.473431 23166 consensus_queue.cc:260] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.474424 23166 ts_tablet_manager.cc:1428] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:15:44.474802 22886 heartbeater.cc:503] Master 127.3.245.62:41131 was elected leader, sending a full tablet report...
I20260620 07:15:44.474752 23167 raft_consensus.cc:348] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.474997 23167 raft_consensus.cc:374] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:44.475059 23167 raft_consensus.cc:729] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259a699c8bad4dd695c145c32d5cd775, State: Initialized, Role: FOLLOWER
I20260620 07:15:44.475492 23167 consensus_queue.cc:260] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.476220 23017 heartbeater.cc:503] Master 127.3.245.62:41131 was elected leader, sending a full tablet report...
I20260620 07:15:44.476465 23167 ts_tablet_manager.cc:1428] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
W20260620 07:15:44.525009 22887 tablet.cc:2369] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:44.539017 23173 raft_consensus.cc:482] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:44.539248 23173 raft_consensus.cc:504] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.540706 23173 leader_election.cc:288] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3ffde88bd2b4373a7ea59b84c871459 (127.3.245.3:40425), 5c636e427bf844f2a3b6e5ae3e52c560 (127.3.245.1:34245)
I20260620 07:15:44.545325 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.540979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:40425 (local address 127.3.245.2:58093)
0620 07:15:44.541168 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.541185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.541308 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.541616 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.541619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.541639 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:44.541759 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.541775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.542820 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.542823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.544034 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.544044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.544897 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.544900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.545001 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.545028 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.545105 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.545183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1}
I20260620 07:15:44.545392 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.541074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:58093 (local address 127.3.245.3:40425)
0620 07:15:44.541192 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.541196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.541244 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:15:44.541423 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.541426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.541479 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.541589 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:44.541907 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.542664 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.544160 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.544792 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.544899 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.544963 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.545136 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.545241 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 07:15:44.545423 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.541389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:41235 (local address 127.3.245.1:34245)
0620 07:15:44.541511 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.541518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.541787 (+ 269us) server_negotiation.cc:407] Connection header received
0620 07:15:44.541910 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.541914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.541966 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.542038 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:44.542354 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.542966 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.544321 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.544927 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.545026 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.545067 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.545246 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.545318 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 07:15:44.545859 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.541272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34245 (local address 127.3.245.2:41235)
0620 07:15:44.541703 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.541719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:44.541821 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:44.542071 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:44.542074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:44.542093 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:44.542209 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.542214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.543074 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.543077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.544199 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:44.544207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.545029 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.545032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:44.545132 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.545150 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 07:15:44.545215 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 07:15:44.545293 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":140,"threads_started":1}
I20260620 07:15:44.546032 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0540047e87884689ab26fa7a5785c16b" candidate_uuid: "259a699c8bad4dd695c145c32d5cd775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ffde88bd2b4373a7ea59b84c871459" is_pre_election: true
I20260620 07:15:44.546382 23102 raft_consensus.cc:2454] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259a699c8bad4dd695c145c32d5cd775 in term 0.
I20260620 07:15:44.546536 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0540047e87884689ab26fa7a5785c16b" candidate_uuid: "259a699c8bad4dd695c145c32d5cd775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" is_pre_election: true
I20260620 07:15:44.546831 22840 raft_consensus.cc:2454] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259a699c8bad4dd695c145c32d5cd775 in term 0.
I20260620 07:15:44.546931 22905 leader_election.cc:302] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259a699c8bad4dd695c145c32d5cd775, d3ffde88bd2b4373a7ea59b84c871459; no voters:
I20260620 07:15:44.547215 23173 raft_consensus.cc:2781] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:44.547353 23173 raft_consensus.cc:482] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:44.547453 23173 raft_consensus.cc:3037] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:44.548823 23173 raft_consensus.cc:504] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.549162 23173 leader_election.cc:288] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [CANDIDATE]: Term 1 election: Requested vote from peers d3ffde88bd2b4373a7ea59b84c871459 (127.3.245.3:40425), 5c636e427bf844f2a3b6e5ae3e52c560 (127.3.245.1:34245)
I20260620 07:15:44.549561 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0540047e87884689ab26fa7a5785c16b" candidate_uuid: "259a699c8bad4dd695c145c32d5cd775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3ffde88bd2b4373a7ea59b84c871459"
I20260620 07:15:44.549569 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0540047e87884689ab26fa7a5785c16b" candidate_uuid: "259a699c8bad4dd695c145c32d5cd775" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c636e427bf844f2a3b6e5ae3e52c560"
I20260620 07:15:44.549710 23102 raft_consensus.cc:3037] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:44.549710 22840 raft_consensus.cc:3037] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:44.551012 23102 raft_consensus.cc:2454] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259a699c8bad4dd695c145c32d5cd775 in term 1.
I20260620 07:15:44.551012 22840 raft_consensus.cc:2454] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259a699c8bad4dd695c145c32d5cd775 in term 1.
I20260620 07:15:44.551350 22905 leader_election.cc:302] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259a699c8bad4dd695c145c32d5cd775, d3ffde88bd2b4373a7ea59b84c871459; no voters:
I20260620 07:15:44.551537 23173 raft_consensus.cc:2781] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:44.551828 23173 raft_consensus.cc:686] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 1 LEADER]: Becoming Leader. State: Replica: 259a699c8bad4dd695c145c32d5cd775, State: Running, Role: LEADER
I20260620 07:15:44.552171 23173 consensus_queue.cc:237] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } }
I20260620 07:15:44.555456 22697 catalog_manager.cc:5495] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259a699c8bad4dd695c145c32d5cd775 (127.3.245.2). New cstate: current_term: 1 leader_uuid: "259a699c8bad4dd695c145c32d5cd775" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259a699c8bad4dd695c145c32d5cd775" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 45263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 } health_report { overall_health: UNKNOWN } } }
I20260620 07:15:44.574572 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.571251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58462 (local address 127.3.245.2:45263)
0620 07:15:44.571386 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:44.571390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:44.571403 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:15:44.571515 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:44.571517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:44.571570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:44.571655 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:44.571864 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.572350 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.573182 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:44.573402 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:44.573478 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:44.573715 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:44.574108 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:44.574217 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:44.574331 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 07:15:44.574389 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:44.580575 23102 raft_consensus.cc:1261] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 1 FOLLOWER]: Refusing update from remote peer 259a699c8bad4dd695c145c32d5cd775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:44.580575 22840 raft_consensus.cc:1261] T 0540047e87884689ab26fa7a5785c16b P 5c636e427bf844f2a3b6e5ae3e52c560 [term 1 FOLLOWER]: Refusing update from remote peer 259a699c8bad4dd695c145c32d5cd775: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:44.581437 23173 consensus_queue.cc:1035] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3ffde88bd2b4373a7ea59b84c871459" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:44.581676 23175 consensus_queue.cc:1035] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c636e427bf844f2a3b6e5ae3e52c560" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:44.589571 23179 mvcc.cc:205] Tried to move back new op lower bound from 7298825193794412544 to 7298825193688915968. Current Snapshot: MvccSnapshot[applied={T|T < 7298825193794412544}]
I20260620 07:15:44.590242 23180 mvcc.cc:205] Tried to move back new op lower bound from 7298825193794412544 to 7298825193688915968. Current Snapshot: MvccSnapshot[applied={T|T < 7298825193794412544}]
I20260620 07:15:44.599433 22698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:44.599613 22698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:44.602486 22698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fd5ddb8613b4027b5b9b1e363737822: Sending DeleteTablet for 3 replicas of tablet 0540047e87884689ab26fa7a5785c16b
I20260620 07:15:44.603415 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 0540047e87884689ab26fa7a5785c16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:44 UTC) from {username='slave'} at 127.0.0.1:58448
I20260620 07:15:44.603420 22820 tablet_service.cc:1514] Processing DeleteTablet for tablet 0540047e87884689ab26fa7a5785c16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:44 UTC) from {username='slave'} at 127.0.0.1:40044
I20260620 07:15:44.604038 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22758
I20260620 07:15:44.604117 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 0540047e87884689ab26fa7a5785c16b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:44 UTC) from {username='slave'} at 127.0.0.1:39646
I20260620 07:15:44.604753 23187 tablet_replica.cc:330] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: stopping tablet replica
I20260620 07:15:44.605074 23187 raft_consensus.cc:2229] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:44.605504 23187 raft_consensus.cc:2258] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:44.607338 23187 ts_tablet_manager.cc:1905] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:44.610092 23187 ts_tablet_manager.cc:1918] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:44.610190 23187 log.cc:1196] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/0540047e87884689ab26fa7a5785c16b
I20260620 07:15:44.610499 23187 ts_tablet_manager.cc:1939] T 0540047e87884689ab26fa7a5785c16b P 259a699c8bad4dd695c145c32d5cd775: Deleting consensus metadata
I20260620 07:15:44.610646 23189 tablet_replica.cc:330] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: stopping tablet replica
I20260620 07:15:44.611055 23189 raft_consensus.cc:2229] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:44.611480 23189 raft_consensus.cc:2258] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 07:15:44.611495 22683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:34245 (error 108)
I20260620 07:15:44.611707 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22889
I20260620 07:15:44.613632 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.613271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:34245 (local address 127.0.0.1:40048)
0620 07:15:44.613450 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.613524 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 07:15:44.613822 22683 catalog_manager.cc:4584] TS 5c636e427bf844f2a3b6e5ae3e52c560 (127.3.245.1:34245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0540047e87884689ab26fa7a5785c16b: Network error: Client connection negotiation failed: client connection to 127.3.245.1:34245: connect: Connection refused (error 111)
I20260620 07:15:44.614674 23189 ts_tablet_manager.cc:1905] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:44.616854 23189 ts_tablet_manager.cc:1918] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:44.616940 23189 log.cc:1196] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/0540047e87884689ab26fa7a5785c16b
I20260620 07:15:44.617238 23189 ts_tablet_manager.cc:1939] T 0540047e87884689ab26fa7a5785c16b P d3ffde88bd2b4373a7ea59b84c871459: Deleting consensus metadata
I20260620 07:15:44.617856 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23020
I20260620 07:15:44.618659 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:44.618130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:45263 (local address 127.0.0.1:58474)
0620 07:15:44.618538 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:44.618590 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:45263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
I20260620 07:15:44.623673 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 22667
2026-06-20T07:15:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:44 dist-test-slave-ldfr krb5kdc[23193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:44 dist-test-slave-ldfr krb5kdc[23193](info): set up 2 sockets
Jun 20 07:15:44 dist-test-slave-ldfr krb5kdc[23193](info): commencing operation
krb5kdc: starting...
W20260620 07:15:45.676822 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:45 dist-test-slave-ldfr krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:15:45.799751 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35829
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:35829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:45.911937 23206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:45.912297 23206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:45.912379 23206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:45.916556 23206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:45.916675 23206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:45.916708 23206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:45.916733 23206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:45.916759 23206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:45.922335 23206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:35829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:45.923470 23206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:45.924518 23206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:45.933197 23214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:45.933205 23212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:45.933558 23206 server_base.cc:1034] running on GCE node
W20260620 07:15:45.933715 23211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:45.934327 23206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:45.935451 23206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:45.936687 23206 hybrid_clock.cc:648] HybridClock initialized: now 1781939745936664 us; error 32 us; skew 500 ppm
Jun 20 07:15:45 dist-test-slave-ldfr krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939745, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:45.939486 23206 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:15:45.940452 23206 webserver.cc:460] Webserver started at http://127.3.245.62:39383/ using document root <none> and password file <none>
I20260620 07:15:45.941071 23206 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:45.941164 23206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:45.941394 23206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:45.943753 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "120d29f84ccd4ed3bb377dab95d2363f"
format_stamp: "Formatted at 2026-06-20 07:15:45 on dist-test-slave-ldfr"
server_key: "fd71ab1b301fd2b5cea0e78fc41b5c33"
server_key_iv: "94fbbd9d51542e0f781f6999fa0876c0"
server_key_version: "encryptionkey@0"
I20260620 07:15:45.944350 23206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "120d29f84ccd4ed3bb377dab95d2363f"
format_stamp: "Formatted at 2026-06-20 07:15:45 on dist-test-slave-ldfr"
server_key: "fd71ab1b301fd2b5cea0e78fc41b5c33"
server_key_iv: "94fbbd9d51542e0f781f6999fa0876c0"
server_key_version: "encryptionkey@0"
I20260620 07:15:45.948218 23206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:15:45.950789 23221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:45.952211 23206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:45.952404 23206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "120d29f84ccd4ed3bb377dab95d2363f"
format_stamp: "Formatted at 2026-06-20 07:15:45 on dist-test-slave-ldfr"
server_key: "fd71ab1b301fd2b5cea0e78fc41b5c33"
server_key_iv: "94fbbd9d51542e0f781f6999fa0876c0"
server_key_version: "encryptionkey@0"
I20260620 07:15:45.952522 23206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:45.977654 23206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:45.981088 23206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:45.981302 23206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:45.988765 23206 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:35829
I20260620 07:15:45.988814 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:35829 every 8 connection(s)
I20260620 07:15:45.989746 23206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:45.992697 23274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:45.996882 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23206
I20260620 07:15:45.997042 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:45.997337 4052 external_mini_cluster.cc:1428] Setting key d75b81311a35f89fe48acda5ee317619
I20260620 07:15:45.998790 23274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: Bootstrap starting.
I20260620 07:15:46.001593 23274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:46.002375 23274 log.cc:824] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:46.004287 23274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: No bootstrap required, opened a new log
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:46.008958 23274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } }
I20260620 07:15:46.009275 23274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:46.009346 23274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120d29f84ccd4ed3bb377dab95d2363f, State: Initialized, Role: FOLLOWER
I20260620 07:15:46.009888 23274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } }
I20260620 07:15:46.010077 23274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:46.010166 23274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:46.010289 23274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:46.011541 23274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } }
I20260620 07:15:46.011909 23274 leader_election.cc:302] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120d29f84ccd4ed3bb377dab95d2363f; no voters:
I20260620 07:15:46.012305 23274 leader_election.cc:288] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:46.012413 23279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:46.012478 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:45.998804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.3.245.62:35829)
0620 07:15:45.999269 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:45.999288 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:45.999330 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:15:46.000217 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.000241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.000685 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.001044 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:46.001746 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.002823 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.003711 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.004184 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.004355 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.007076 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:46.007083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:46.007108 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:46.007146 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:46.009246 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.009780 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.009787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.009793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.009911 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.010216 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.010218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.010219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.010720 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:46.010877 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.011195 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.011554 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":168,"threads_started":1}
I20260620 07:15:46.013475 23279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [term 1 LEADER]: Becoming Leader. State: Replica: 120d29f84ccd4ed3bb377dab95d2363f, State: Running, Role: LEADER
I20260620 07:15:46.013662 23274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:46.013783 23279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } }
I20260620 07:15:46.015969 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "120d29f84ccd4ed3bb377dab95d2363f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } } }
I20260620 07:15:46.016014 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120d29f84ccd4ed3bb377dab95d2363f. Latest consensus state: current_term: 1 leader_uuid: "120d29f84ccd4ed3bb377dab95d2363f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120d29f84ccd4ed3bb377dab95d2363f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35829 } } }
I20260620 07:15:46.016355 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:46.016356 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f [sys.catalog]: This master's current role is: LEADER
W20260620 07:15:46.018888 23293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:46.018972 23293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:46.019032 23294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:46.019986 23294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:46.025698 23294 catalog_manager.cc:1347] Generated new cluster ID: 4f35b30549e946f5b8ec0fe109f10052
I20260620 07:15:46.025771 23294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:46.043956 23294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:46.044979 23294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:46.051983 23294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: Generated new TSK 0
I20260620 07:15:46.052822 23294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.245.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
Entry for principal kudu/127.3.245.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab.
I20260620 07:15:46.085825 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:46.199048 23300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:46.199321 23300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:46.199365 23300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:46.203521 23300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:46.203595 23300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:46.203655 23300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:46.209034 23300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.1.keytab
--principal=kudu/127.3.245.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:46.210031 23300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:46.211043 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:46.220758 23305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.220890 23300 server_base.cc:1034] running on GCE node
W20260620 07:15:46.220770 23306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:46.221058 23308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.221448 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:46.222069 23300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:46.223256 23300 hybrid_clock.cc:648] HybridClock initialized: now 1781939746223235 us; error 28 us; skew 500 ppm
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:46.225927 23300 init.cc:377] Logged in from keytab as kudu/127.3.245.1@KRBTEST.COM (short username kudu)
I20260620 07:15:46.226806 23300 webserver.cc:460] Webserver started at http://127.3.245.1:35879/ using document root <none> and password file <none>
I20260620 07:15:46.227367 23300 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:46.227456 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:46.227681 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:46.229722 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "7d37dbc6ea724c6db078bc4567aa3862"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "e5b185345c2fb31836ff86294010851e"
server_key_iv: "51b82b795d8a79c08bc68190be39c8ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.230288 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "7d37dbc6ea724c6db078bc4567aa3862"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "e5b185345c2fb31836ff86294010851e"
server_key_iv: "51b82b795d8a79c08bc68190be39c8ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.233971 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:15:46.236243 23315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.237488 23300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:46.237619 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "7d37dbc6ea724c6db078bc4567aa3862"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "e5b185345c2fb31836ff86294010851e"
server_key_iv: "51b82b795d8a79c08bc68190be39c8ad"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.237728 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:46.261976 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:46.264988 23300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:46.265202 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:46.265815 23300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:46.266669 23300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:46.266739 23300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.266808 23300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:46.266850 23300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.276216 23300 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:45321
I20260620 07:15:46.276234 23428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:45321 every 8 connection(s)
I20260620 07:15:46.277184 23300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:46.282521 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23300
I20260620 07:15:46.282666 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:46.282945 4052 external_mini_cluster.cc:1428] Setting key cf9baf1e760599321cd5ac036a3aaf34
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.1@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:46.291236 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.279289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:59963 (local address 127.3.245.62:35829)
0620 07:15:46.279427 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.279431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.280304 (+ 873us) server_negotiation.cc:407] Connection header received
0620 07:15:46.281301 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.281305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.281362 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.281441 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:46.282636 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.283313 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.284658 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.284902 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.285013 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.287687 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:46.287689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:46.287692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:46.287715 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:46.289210 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.289734 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.289738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.289740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.289829 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.290167 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.290172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.290174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.290355 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:46.290459 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.290953 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.291075 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.3.245.2@KRBTEST.COM; defaulting to no policy
I20260620 07:15:46.292136 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.279680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35829 (local address 127.3.245.1:59963)
0620 07:15:46.280126 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.280168 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.281063 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.281628 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.281634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.281999 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:46.282483 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.282497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.283440 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.283444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.284523 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.284532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.285048 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.285053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.285217 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.285827 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:46.285844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:46.287507 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:46.289346 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.289352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.289367 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.289629 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.289923 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.289927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.289929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.290073 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.290458 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:46.290467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:46.290722 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.290914 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.291231 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260620 07:15:46.293483 23429 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35829
I20260620 07:15:46.293828 23429 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.2@KRBTEST.COM" created.
I20260620 07:15:46.294440 23429 heartbeater.cc:511] Master 127.3.245.62:35829 requested a full tablet report, sending...
I20260620 07:15:46.296293 23238 ts_manager.cc:194] Registered new tserver with Master: 7d37dbc6ea724c6db078bc4567aa3862 (127.3.245.1:45321)
I20260620 07:15:46.297508 23238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.1@KRBTEST.COM'} at 127.3.245.1:59963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
Entry for principal kudu/127.3.245.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab.
I20260620 07:15:46.311348 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:46.425459 23434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:46.425729 23434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:46.425772 23434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:46.429755 23434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:46.429822 23434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:46.429880 23434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:46.435160 23434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.2.keytab
--principal=kudu/127.3.245.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:46.436189 23434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:46.437237 23434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:46.447070 23440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:46.447081 23439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.447147 23434 server_base.cc:1034] running on GCE node
W20260620 07:15:46.447081 23442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.447667 23434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:46.448280 23434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:46.449448 23434 hybrid_clock.cc:648] HybridClock initialized: now 1781939746449425 us; error 46 us; skew 500 ppm
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:46.452308 23434 init.cc:377] Logged in from keytab as kudu/127.3.245.2@KRBTEST.COM (short username kudu)
I20260620 07:15:46.453316 23434 webserver.cc:460] Webserver started at http://127.3.245.2:38409/ using document root <none> and password file <none>
I20260620 07:15:46.453832 23434 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:46.453887 23434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:46.454047 23434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:46.456096 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "65150ca590cb420b93a2ea378d003013"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "d774e36ba92a8289e0843b25cf465555"
server_key_iv: "da06fe8631687c48c708121b4c076d27"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.456590 23434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "65150ca590cb420b93a2ea378d003013"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "d774e36ba92a8289e0843b25cf465555"
server_key_iv: "da06fe8631687c48c708121b4c076d27"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.460196 23434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:15:46.462517 23449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.463694 23434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:46.463838 23434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "65150ca590cb420b93a2ea378d003013"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "d774e36ba92a8289e0843b25cf465555"
server_key_iv: "da06fe8631687c48c708121b4c076d27"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.463939 23434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:46.481493 23434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:46.485092 23434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:46.485303 23434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:46.485956 23434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:46.486783 23434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:46.486829 23434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.486898 23434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:46.486943 23434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.496155 23434 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:41231
I20260620 07:15:46.496177 23562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:41231 every 8 connection(s)
I20260620 07:15:46.497072 23434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:46.498334 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23434
I20260620 07:15:46.498483 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:46.498795 4052 external_mini_cluster.cc:1428] Setting key fd5ec9418300a8a3caae110fe56c7f7f
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.2@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.245.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.3@KRBTEST.COM" created.
I20260620 07:15:46.513387 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.499219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:34283 (local address 127.3.245.62:35829)
0620 07:15:46.499437 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.499445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.500454 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 07:15:46.501539 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.501546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.501612 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.501714 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:46.503461 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.503989 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.505911 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.506130 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.506213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.509269 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:46.509273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:46.509276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:46.509315 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:46.510747 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.511439 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.511441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.511443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.511517 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.512036 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.512043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.512048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.512287 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:46.512367 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.513136 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.513244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:15:46.514760 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.499662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35829 (local address 127.3.245.2:34283)
0620 07:15:46.500239 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.500320 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.501330 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.501954 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.501965 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.502503 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:46.503283 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.503321 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.504164 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.504171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.505774 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.505789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.506302 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.506308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.506560 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.507197 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:46.507218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:46.509053 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:46.510883 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.510888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.510901 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.511337 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.511699 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.511703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.511706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.511923 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.512409 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:46.512418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:46.512868 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.513081 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.513458 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
I20260620 07:15:46.516288 23563 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35829
I20260620 07:15:46.516582 23563 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:46.517154 23563 heartbeater.cc:511] Master 127.3.245.62:35829 requested a full tablet report, sending...
I20260620 07:15:46.518280 23238 ts_manager.cc:194] Registered new tserver with Master: 65150ca590cb420b93a2ea378d003013 (127.3.245.2:41231)
I20260620 07:15:46.518940 23238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.2@KRBTEST.COM'} at 127.3.245.2:34283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
Entry for principal kudu/127.3.245.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab.
I20260620 07:15:46.527906 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:46.640331 23568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:46.640712 23568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:46.640808 23568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:46.645205 23568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:46.645301 23568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:46.645393 23568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:46.650770 23568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.3.keytab
--principal=kudu/127.3.245.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:46.651855 23568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:46.652984 23568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:46.662525 23576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:46.662585 23574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.662706 23568 server_base.cc:1034] running on GCE node
W20260620 07:15:46.662563 23573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:46.663239 23568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:46.663934 23568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:46.665091 23568 hybrid_clock.cc:648] HybridClock initialized: now 1781939746665076 us; error 29 us; skew 500 ppm
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:46.667834 23568 init.cc:377] Logged in from keytab as kudu/127.3.245.3@KRBTEST.COM (short username kudu)
I20260620 07:15:46.668788 23568 webserver.cc:460] Webserver started at http://127.3.245.3:37541/ using document root <none> and password file <none>
I20260620 07:15:46.669355 23568 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:46.669415 23568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:46.669618 23568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:46.671597 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "0f050493f20b48ebb9fee6e0a8c788f6"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "6a790ff89b0ca804170e99f2d1b2280c"
server_key_iv: "dcb26b657f51f9bc9777e59babaeb043"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.672139 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "0f050493f20b48ebb9fee6e0a8c788f6"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "6a790ff89b0ca804170e99f2d1b2280c"
server_key_iv: "dcb26b657f51f9bc9777e59babaeb043"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.675823 23568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 07:15:46.678390 23583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.679576 23568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:15:46.679705 23568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "0f050493f20b48ebb9fee6e0a8c788f6"
format_stamp: "Formatted at 2026-06-20 07:15:46 on dist-test-slave-ldfr"
server_key: "6a790ff89b0ca804170e99f2d1b2280c"
server_key_iv: "dcb26b657f51f9bc9777e59babaeb043"
server_key_version: "encryptionkey@0"
I20260620 07:15:46.679816 23568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:46.704461 23568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:46.707787 23568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:46.707993 23568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:46.708598 23568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:46.709498 23568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:46.709543 23568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.709614 23568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:46.709652 23568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:46.718828 23568 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:39153
I20260620 07:15:46.718856 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:39153 every 8 connection(s)
I20260620 07:15:46.719707 23568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:46.724658 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23568
I20260620 07:15:46.724809 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:46.725103 4052 external_mini_cluster.cc:1428] Setting key 405325d2b126822e3d24b3d8fb980226
Jun 20 07:15:46 dist-test-slave-ldfr krb5kdc[23193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939746, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.3@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:15:46.733485 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.721601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:48609 (local address 127.3.245.62:35829)
0620 07:15:46.721738 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.721742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.722833 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 07:15:46.723774 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.723777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.723835 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.723917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:46.725285 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.725927 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.727182 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.727412 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.727525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.729948 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:46.729952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:46.729955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:46.729987 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:46.731427 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.731944 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.731949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.731951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.732035 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.732380 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.732384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.732386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.732569 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:46.732711 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.733195 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.733316 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:15:46.734366 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.721987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:35829 (local address 127.3.245.3:48609)
0620 07:15:46.722672 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.722716 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.723561 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.724089 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.724098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.724475 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:46.725110 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.725129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.726040 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.726044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.727051 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.727061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.727560 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.727564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.727736 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.728336 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:15:46.728352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:46.729780 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:46.731559 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.731566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.731580 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.731842 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.732135 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:15:46.732138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:15:46.732140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:15:46.732284 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:15:46.732711 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:46.732719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:46.732958 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.733143 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.733466 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":107,"threads_started":1}
I20260620 07:15:46.735476 23697 heartbeater.cc:347] Connected to a master server at 127.3.245.62:35829
I20260620 07:15:46.735716 23697 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:46.736223 23697 heartbeater.cc:511] Master 127.3.245.62:35829 requested a full tablet report, sending...
I20260620 07:15:46.737275 23238 ts_manager.cc:194] Registered new tserver with Master: 0f050493f20b48ebb9fee6e0a8c788f6 (127.3.245.3:39153)
I20260620 07:15:46.737967 23238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.245.3@KRBTEST.COM'} at 127.3.245.3:48609
I20260620 07:15:46.739097 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:46.749109 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.741122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.3.245.62:35829)
0620 07:15:46.741227 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.741232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.741461 (+ 229us) server_negotiation.cc:407] Connection header received
0620 07:15:46.741566 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.741569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.741610 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.741699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:46.742505 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.743017 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.743891 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.744139 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.744244 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.745824 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:46.745830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:46.745837 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:46.745860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:46.747394 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.747823 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.747828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.747830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.747917 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:46.748185 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:46.748189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:46.748191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:46.748413 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:46.748551 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.748834 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.748950 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 07:15:46.751933 23238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:46.754977 23238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:46.769490 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.763630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:41231 (local address 127.0.0.1:44916)
0620 07:15:46.764437 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.764450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.764535 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.765340 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.765342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.765357 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:46.765463 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.765470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766841 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.766845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.768039 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.768050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.769074 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.769078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.769162 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.769183 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.769255 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.769331 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":81,"threads_started":1}
I20260620 07:15:46.769528 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.764010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.3.245.2:41231)
0620 07:15:46.764728 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.764739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.764758 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:15:46.764830 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.764836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.765017 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.765160 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:46.765597 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766678 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.768145 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.768969 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.769074 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.769200 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.769292 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.769386 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":133,"threads_started":1}
I20260620 07:15:46.769670 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.763395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.3.245.1:45321)
0620 07:15:46.764459 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.764477 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.764741 (+ 264us) server_negotiation.cc:407] Connection header received
0620 07:15:46.764831 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.764836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.765017 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.765160 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:46.765434 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766231 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.767772 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.768707 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.768826 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.768981 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.769414 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.769511 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":124,"threads_started":1}
I20260620 07:15:46.769538 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.763401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:39153 (local address 127.0.0.1:46820)
0620 07:15:46.763995 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.764041 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.764184 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.765029 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.765035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.765057 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:46.765196 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.765203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766707 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.766711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.768117 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.768127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.769185 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.769188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.769268 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.769287 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.769354 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.769418 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":217,"thread_start_us":101,"threads_started":1}
I20260620 07:15:46.770267 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.763997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.3.245.3:39153)
0620 07:15:46.764463 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:46.764479 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:46.764503 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:15:46.764583 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:46.764590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:46.764880 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:46.765040 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:46.765336 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766558 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.768218 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.768887 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.768985 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.769125 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:46.770015 (+ 890us) server_negotiation.cc:299] Negotiation successful
0620 07:15:46.770123 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":137,"threads_started":1}
I20260620 07:15:46.770273 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:46.763228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:45321 (local address 127.0.0.1:37674)
0620 07:15:46.764633 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:46.764646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:46.764733 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:46.765195 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:46.765198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:46.765213 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:46.765321 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.765327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.766360 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.766366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.767643 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:46.767652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:46.768858 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:46.768861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:46.768985 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:46.769029 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:15:46.769138 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 07:15:46.769279 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":6,"thread_start_us":122,"threads_started":1}
I20260620 07:15:46.772814 23631 tablet_service.cc:1467] Processing CreateTablet for tablet f387a284364142eb9cf0e084772be6ae (DEFAULT_TABLE table=test-table [id=55bbb8ce886744e88bb20cd3d36a942c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:46.772881 23497 tablet_service.cc:1467] Processing CreateTablet for tablet f387a284364142eb9cf0e084772be6ae (DEFAULT_TABLE table=test-table [id=55bbb8ce886744e88bb20cd3d36a942c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:46.773029 23363 tablet_service.cc:1467] Processing CreateTablet for tablet f387a284364142eb9cf0e084772be6ae (DEFAULT_TABLE table=test-table [id=55bbb8ce886744e88bb20cd3d36a942c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:46.773872 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387a284364142eb9cf0e084772be6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:46.773903 23497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387a284364142eb9cf0e084772be6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:46.773903 23363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f387a284364142eb9cf0e084772be6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:46.780067 23714 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Bootstrap starting.
I20260620 07:15:46.780813 23715 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Bootstrap starting.
I20260620 07:15:46.782483 23716 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Bootstrap starting.
I20260620 07:15:46.782744 23714 tablet_bootstrap.cc:654] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:46.783159 23715 tablet_bootstrap.cc:654] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:46.784101 23715 log.cc:824] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:46.784101 23714 log.cc:824] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:46.784601 23716 tablet_bootstrap.cc:654] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:46.785820 23716 log.cc:824] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:46.785992 23715 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: No bootstrap required, opened a new log
I20260620 07:15:46.786202 23714 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: No bootstrap required, opened a new log
I20260620 07:15:46.786226 23715 ts_tablet_manager.cc:1397] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 07:15:46.786427 23714 ts_tablet_manager.cc:1397] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 07:15:46.787655 23716 tablet_bootstrap.cc:492] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: No bootstrap required, opened a new log
I20260620 07:15:46.787904 23716 ts_tablet_manager.cc:1397] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 07:15:46.789492 23715 raft_consensus.cc:348] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.789706 23715 raft_consensus.cc:374] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:46.789791 23715 raft_consensus.cc:729] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65150ca590cb420b93a2ea378d003013, State: Initialized, Role: FOLLOWER
I20260620 07:15:46.790242 23715 consensus_queue.cc:260] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.790311 23714 raft_consensus.cc:348] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.790580 23714 raft_consensus.cc:374] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:46.790647 23714 raft_consensus.cc:729] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f050493f20b48ebb9fee6e0a8c788f6, State: Initialized, Role: FOLLOWER
I20260620 07:15:46.790926 23563 heartbeater.cc:503] Master 127.3.245.62:35829 was elected leader, sending a full tablet report...
I20260620 07:15:46.791245 23715 ts_tablet_manager.cc:1428] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260620 07:15:46.791646 23714 consensus_queue.cc:260] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.792104 23716 raft_consensus.cc:348] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.792356 23697 heartbeater.cc:503] Master 127.3.245.62:35829 was elected leader, sending a full tablet report...
I20260620 07:15:46.792382 23716 raft_consensus.cc:374] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:46.792510 23716 raft_consensus.cc:729] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d37dbc6ea724c6db078bc4567aa3862, State: Initialized, Role: FOLLOWER
I20260620 07:15:46.792637 23714 ts_tablet_manager.cc:1428] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 07:15:46.793073 23716 consensus_queue.cc:260] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:46.793776 23429 heartbeater.cc:503] Master 127.3.245.62:35829 was elected leader, sending a full tablet report...
I20260620 07:15:46.794270 23716 ts_tablet_manager.cc:1428] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260620 07:15:46.970664 23698 tablet.cc:2369] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:46.998422 23564 tablet.cc:2369] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:15:47.028784 23430 tablet.cc:2369] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:47.161592 23722 raft_consensus.cc:482] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:47.161795 23722 raft_consensus.cc:504] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:47.162947 23722 leader_election.cc:288] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f050493f20b48ebb9fee6e0a8c788f6 (127.3.245.3:39153), 65150ca590cb420b93a2ea378d003013 (127.3.245.2:41231)
I20260620 07:15:47.166921 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.163318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:44053 (local address 127.3.245.2:41231)
0620 07:15:47.163459 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.163463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.163602 (+ 139us) server_negotiation.cc:407] Connection header received
0620 07:15:47.163820 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.163822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.163867 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.163953 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:47.164329 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.164919 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.165969 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.166369 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.166488 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.166552 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.166718 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.166821 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:15:47.166945 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.163169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:41231 (local address 127.3.245.1:44053)
0620 07:15:47.163488 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.163506 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:47.163659 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:47.163964 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:47.163967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:47.163984 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:47.164155 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.164165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.165050 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.165052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.165847 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.165854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.166493 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.166497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.166584 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.166614 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:15:47.166703 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:15:47.166776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"spinlock_wait_cycles":3712,"thread_start_us":121,"threads_started":1}
I20260620 07:15:47.167126 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.163169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:39153 (local address 127.3.245.1:58981)
0620 07:15:47.163877 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.163889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:47.164017 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:47.164309 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:47.164312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:47.164328 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:47.164442 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.164447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.165224 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.165227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.166012 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.166020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.166660 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.166664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.166781 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.166824 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 07:15:47.166900 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:15:47.166970 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":312,"thread_start_us":152,"threads_started":1}
I20260620 07:15:47.167109 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.163317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:58981 (local address 127.3.245.3:39153)
0620 07:15:47.163466 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.163470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.163985 (+ 515us) server_negotiation.cc:407] Connection header received
0620 07:15:47.164133 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.164136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.164187 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.164275 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:47.164551 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.165104 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.166134 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.166534 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.166645 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.166707 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.166917 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.166987 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:15:47.167577 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387a284364142eb9cf0e084772be6ae" candidate_uuid: "7d37dbc6ea724c6db078bc4567aa3862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65150ca590cb420b93a2ea378d003013" is_pre_election: true
I20260620 07:15:47.167776 23651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387a284364142eb9cf0e084772be6ae" candidate_uuid: "7d37dbc6ea724c6db078bc4567aa3862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" is_pre_election: true
I20260620 07:15:47.167918 23517 raft_consensus.cc:2454] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d37dbc6ea724c6db078bc4567aa3862 in term 0.
I20260620 07:15:47.168078 23651 raft_consensus.cc:2454] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d37dbc6ea724c6db078bc4567aa3862 in term 0.
I20260620 07:15:47.168454 23318 leader_election.cc:302] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65150ca590cb420b93a2ea378d003013, 7d37dbc6ea724c6db078bc4567aa3862; no voters:
I20260620 07:15:47.168797 23722 raft_consensus.cc:2781] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:47.168897 23722 raft_consensus.cc:482] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:47.168959 23722 raft_consensus.cc:3037] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:47.169996 23722 raft_consensus.cc:504] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:47.170369 23722 leader_election.cc:288] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [CANDIDATE]: Term 1 election: Requested vote from peers 0f050493f20b48ebb9fee6e0a8c788f6 (127.3.245.3:39153), 65150ca590cb420b93a2ea378d003013 (127.3.245.2:41231)
I20260620 07:15:47.170771 23517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387a284364142eb9cf0e084772be6ae" candidate_uuid: "7d37dbc6ea724c6db078bc4567aa3862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65150ca590cb420b93a2ea378d003013"
I20260620 07:15:47.170912 23517 raft_consensus.cc:3037] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:47.170950 23651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f387a284364142eb9cf0e084772be6ae" candidate_uuid: "7d37dbc6ea724c6db078bc4567aa3862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f050493f20b48ebb9fee6e0a8c788f6"
I20260620 07:15:47.171100 23651 raft_consensus.cc:3037] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:47.172442 23517 raft_consensus.cc:2454] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d37dbc6ea724c6db078bc4567aa3862 in term 1.
I20260620 07:15:47.172796 23651 raft_consensus.cc:2454] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d37dbc6ea724c6db078bc4567aa3862 in term 1.
I20260620 07:15:47.172946 23318 leader_election.cc:302] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65150ca590cb420b93a2ea378d003013, 7d37dbc6ea724c6db078bc4567aa3862; no voters:
I20260620 07:15:47.175202 23722 raft_consensus.cc:2781] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:47.175470 23722 raft_consensus.cc:686] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 1 LEADER]: Becoming Leader. State: Replica: 7d37dbc6ea724c6db078bc4567aa3862, State: Running, Role: LEADER
I20260620 07:15:47.175927 23722 consensus_queue.cc:237] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } }
I20260620 07:15:47.179715 23237 catalog_manager.cc:5495] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d37dbc6ea724c6db078bc4567aa3862 (127.3.245.1). New cstate: current_term: 1 leader_uuid: "7d37dbc6ea724c6db078bc4567aa3862" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d37dbc6ea724c6db078bc4567aa3862" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 } health_report { overall_health: UNKNOWN } } }
I20260620 07:15:47.223369 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.219591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.3.245.1:45321)
0620 07:15:47.219716 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.219719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.219735 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:15:47.219897 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.219899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.219953 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.220043 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:47.220313 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.220916 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.221933 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.222160 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.222242 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.222525 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:47.222985 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:47.223076 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.223186 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.223246 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:15:47.229363 23517 raft_consensus.cc:1261] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 1 FOLLOWER]: Refusing update from remote peer 7d37dbc6ea724c6db078bc4567aa3862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:47.229370 23651 raft_consensus.cc:1261] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 1 FOLLOWER]: Refusing update from remote peer 7d37dbc6ea724c6db078bc4567aa3862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:47.230122 23725 consensus_queue.cc:1035] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f050493f20b48ebb9fee6e0a8c788f6" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:47.230311 23722 consensus_queue.cc:1035] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65150ca590cb420b93a2ea378d003013" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 41231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:47.240429 23728 mvcc.cc:205] Tried to move back new op lower bound from 7298825204644093952 to 7298825204435894272. Current Snapshot: MvccSnapshot[applied={T|T < 7298825204644093952 or (T in {7298825204644093952})}]
I20260620 07:15:47.247165 23237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:47.247367 23237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:47.249944 23237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 120d29f84ccd4ed3bb377dab95d2363f: Sending DeleteTablet for 3 replicas of tablet f387a284364142eb9cf0e084772be6ae
I20260620 07:15:47.250772 23363 tablet_service.cc:1514] Processing DeleteTablet for tablet f387a284364142eb9cf0e084772be6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:47 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:37674
I20260620 07:15:47.250813 23631 tablet_service.cc:1514] Processing DeleteTablet for tablet f387a284364142eb9cf0e084772be6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:47 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:46820
I20260620 07:15:47.250803 23497 tablet_service.cc:1514] Processing DeleteTablet for tablet f387a284364142eb9cf0e084772be6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:47 UTC) from {username='kudu', principal='kudu/127.3.245.62@KRBTEST.COM'} at 127.0.0.1:44916
I20260620 07:15:47.251281 23739 tablet_replica.cc:330] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862: stopping tablet replica
I20260620 07:15:47.251606 23739 raft_consensus.cc:2229] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:47.251698 23740 tablet_replica.cc:330] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013: stopping tablet replica
I20260620 07:15:47.251979 23739 raft_consensus.cc:2258] T f387a284364142eb9cf0e084772be6ae P 7d37dbc6ea724c6db078bc4567aa3862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:47.251986 23740 raft_consensus.cc:2229] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:47.252342 23738 tablet_replica.cc:330] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: stopping tablet replica
I20260620 07:15:47.252357 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23300
I20260620 07:15:47.253208 23738 raft_consensus.cc:2229] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:47.253515 23738 raft_consensus.cc:2258] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:47.255318 23738 ts_tablet_manager.cc:1905] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:47.258730 23738 ts_tablet_manager.cc:1918] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:47.258890 23738 log.cc:1196] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/f387a284364142eb9cf0e084772be6ae
I20260620 07:15:47.252440 23740 raft_consensus.cc:2258] T f387a284364142eb9cf0e084772be6ae P 65150ca590cb420b93a2ea378d003013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:47.259212 23738 ts_tablet_manager.cc:1939] T f387a284364142eb9cf0e084772be6ae P 0f050493f20b48ebb9fee6e0a8c788f6: Deleting consensus metadata
W20260620 07:15:47.259194 23222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:45321 (error 108)
I20260620 07:15:47.259980 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23434
I20260620 07:15:47.260739 23223 catalog_manager.cc:4841] TS 0f050493f20b48ebb9fee6e0a8c788f6 (127.3.245.3:39153): tablet f387a284364142eb9cf0e084772be6ae (table test-table [id=55bbb8ce886744e88bb20cd3d36a942c]) successfully deleted
I20260620 07:15:47.262110 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.261842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:45321 (local address 127.0.0.1:37680)
0620 07:15:47.261973 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.262038 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:45321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:15:47.262300 23222 catalog_manager.cc:4584] TS 7d37dbc6ea724c6db078bc4567aa3862 (127.3.245.1:45321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f387a284364142eb9cf0e084772be6ae: Network error: Client connection negotiation failed: client connection to 127.3.245.1:45321: connect: Connection refused (error 111)
I20260620 07:15:47.266898 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23568
I20260620 07:15:47.267661 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.266998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:41231 (local address 127.0.0.1:44920)
0620 07:15:47.267542 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.267590 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.2:41231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
I20260620 07:15:47.273373 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23206
2026-06-20T07:15:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2651 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T07:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:47Z Disabled control of system clock
I20260620 07:15:47.305351 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:43295
--webserver_interface=127.3.245.62
--webserver_port=0
--builtin_ntp_servers=127.3.245.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:47.418057 23745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:47.418345 23745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:47.418381 23745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:47.422391 23745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:47.422457 23745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:47.422473 23745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:47.422490 23745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:47.422505 23745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:47.427387 23745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:43295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:47.428481 23745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:47.429940 23745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:47.438444 23753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.438482 23751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.438437 23750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:47.438832 23745 server_base.cc:1034] running on GCE node
I20260620 07:15:47.439415 23745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:47.440541 23745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:47.441702 23745 hybrid_clock.cc:648] HybridClock initialized: now 1781939747441665 us; error 53 us; skew 500 ppm
I20260620 07:15:47.443652 23745 webserver.cc:460] Webserver started at http://127.3.245.62:34681/ using document root <none> and password file <none>
I20260620 07:15:47.444234 23745 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:47.444304 23745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:47.444536 23745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:47.446722 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "f7bb59dd76aa4d8dee76f8ce2e144db0"
server_key_iv: "723206f8f472c51da6d660bc2a74f2c4"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.447302 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "f7bb59dd76aa4d8dee76f8ce2e144db0"
server_key_iv: "723206f8f472c51da6d660bc2a74f2c4"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.451321 23745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:47.454018 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.455385 23745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:15:47.455493 23745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "f7bb59dd76aa4d8dee76f8ce2e144db0"
server_key_iv: "723206f8f472c51da6d660bc2a74f2c4"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.455592 23745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:47.478297 23745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:47.479005 23745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:47.479164 23745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:47.486814 23745 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:43295
I20260620 07:15:47.486824 23811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:43295 every 8 connection(s)
I20260620 07:15:47.487818 23745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:47.491075 23812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:47.491397 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23745
I20260620 07:15:47.491540 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:47.491887 4052 external_mini_cluster.cc:1428] Setting key dd9173f75c8067a7c45cd2e4043e679a
I20260620 07:15:47.497830 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: Bootstrap starting.
I20260620 07:15:47.500756 23812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:47.501626 23812 log.cc:824] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:47.502290 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.493443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.3.245.62:43295)
0620 07:15:47.494108 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.494123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.494160 (+ 37us) server_negotiation.cc:407] Connection header received
0620 07:15:47.494840 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.494858 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.495273 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.495555 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:47.496037 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.497189 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.498372 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.498868 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.498989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.499989 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:47.499999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:47.500018 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:47.500639 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:47.500737 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:47.500770 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:47.500881 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.501117 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.501425 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":163,"threads_started":1}
I20260620 07:15:47.503880 23812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: No bootstrap required, opened a new log
I20260620 07:15:47.507799 23812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } }
I20260620 07:15:47.508041 23812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:47.508102 23812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b60c7c67d5e4c2a8aa84d11082f9d0c, State: Initialized, Role: FOLLOWER
I20260620 07:15:47.508540 23812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } }
I20260620 07:15:47.508765 23812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:47.508852 23812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:47.508958 23812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:47.509955 23812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } }
I20260620 07:15:47.510337 23812 leader_election.cc:302] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b60c7c67d5e4c2a8aa84d11082f9d0c; no voters:
I20260620 07:15:47.510669 23812 leader_election.cc:288] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:47.510805 23817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:47.511052 23817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [term 1 LEADER]: Becoming Leader. State: Replica: 1b60c7c67d5e4c2a8aa84d11082f9d0c, State: Running, Role: LEADER
I20260620 07:15:47.511422 23817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } }
I20260620 07:15:47.511909 23812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:47.513227 23819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b60c7c67d5e4c2a8aa84d11082f9d0c. Latest consensus state: current_term: 1 leader_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } } }
I20260620 07:15:47.513355 23819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:47.513631 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b60c7c67d5e4c2a8aa84d11082f9d0c" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 43295 } } }
I20260620 07:15:47.513754 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:47.514000 23826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:47.516860 23826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:47.522581 23826 catalog_manager.cc:1347] Generated new cluster ID: 1260eb37c6164b88aac63898fd0aed36
I20260620 07:15:47.522658 23826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:47.540151 23826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:47.541229 23826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:47.547639 23826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: Generated new TSK 0
I20260620 07:15:47.548409 23826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:47.554576 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.1:0
--local_ip_for_outbound_sockets=127.3.245.1
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--builtin_ntp_servers=127.3.245.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:15:47.667829 23836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:47.668128 23836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:47.668200 23836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:47.672282 23836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:47.672379 23836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:47.672472 23836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.1
I20260620 07:15:47.677457 23836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.245.1
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.1
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:47.678620 23836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:47.680088 23836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:47.690016 23842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.690021 23844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.690011 23841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:47.690455 23836 server_base.cc:1034] running on GCE node
I20260620 07:15:47.690847 23836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:47.691497 23836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:47.692694 23836 hybrid_clock.cc:648] HybridClock initialized: now 1781939747692647 us; error 55 us; skew 500 ppm
I20260620 07:15:47.700937 23836 webserver.cc:460] Webserver started at http://127.3.245.1:37561/ using document root <none> and password file <none>
I20260620 07:15:47.701561 23836 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:47.701629 23836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:47.701808 23836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:47.703972 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data/instance:
uuid: "4c762c4445a7466497710857efa6cc0f"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "a67fe3da3417dd85c600c654fc7f12b2"
server_key_iv: "d5d2afc2a39b074f743cf4f4d075ddf6"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.704520 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance:
uuid: "4c762c4445a7466497710857efa6cc0f"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "a67fe3da3417dd85c600c654fc7f12b2"
server_key_iv: "d5d2afc2a39b074f743cf4f4d075ddf6"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.708341 23836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:15:47.710894 23850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.712162 23836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:47.712352 23836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal
uuid: "4c762c4445a7466497710857efa6cc0f"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "a67fe3da3417dd85c600c654fc7f12b2"
server_key_iv: "d5d2afc2a39b074f743cf4f4d075ddf6"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.712455 23836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:47.736241 23836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:47.737011 23836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:47.737172 23836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:47.737722 23836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:47.738643 23836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:47.738691 23836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.738731 23836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:47.738754 23836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.747583 23836 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.1:45901
I20260620 07:15:47.747610 23963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.1:45901 every 8 connection(s)
I20260620 07:15:47.748550 23836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/data/info.pb
I20260620 07:15:47.751509 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23836
I20260620 07:15:47.751639 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-0/wal/instance
I20260620 07:15:47.751977 4052 external_mini_cluster.cc:1428] Setting key 8c55c9f01e3df7afec2aec7ed6553898
I20260620 07:15:47.754087 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.2:0
--local_ip_for_outbound_sockets=127.3.245.2
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--builtin_ntp_servers=127.3.245.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:47.759362 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.750410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.1:59215 (local address 127.3.245.62:43295)
0620 07:15:47.750545 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.750549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.751795 (+ 1246us) server_negotiation.cc:407] Connection header received
0620 07:15:47.753550 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.753553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.753596 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.753663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:47.754902 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.755536 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.756985 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.757405 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.757532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.758773 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:47.758780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:47.758786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:47.758803 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:47.758831 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:47.758846 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:47.758926 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.759078 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.759156 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:15:47.760195 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.750831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43295 (local address 127.3.245.1:59215)
0620 07:15:47.751594 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.751657 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:47.753288 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:47.753843 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:47.753854 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:47.753971 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:47.754753 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.754774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.755675 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.755678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.756861 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.756871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.757549 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.757552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.757797 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.758470 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:47.758472 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:47.758513 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:47.758606 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:47.758941 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:47.758948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:47.758952 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 07:15:47.759042 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:15:47.759263 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260620 07:15:47.761545 23964 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43295
I20260620 07:15:47.761824 23964 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:47.762511 23964 heartbeater.cc:511] Master 127.3.245.62:43295 requested a full tablet report, sending...
I20260620 07:15:47.764245 23776 ts_manager.cc:194] Registered new tserver with Master: 4c762c4445a7466497710857efa6cc0f (127.3.245.1:45901)
I20260620 07:15:47.765506 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.1:59215
W20260620 07:15:47.868312 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:47.868646 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:47.868687 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:47.872970 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:47.873039 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:47.873129 23967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.2
I20260620 07:15:47.878650 23967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.245.2
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.2
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:47.879803 23967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:47.881315 23967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:47.891518 23972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.891528 23975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:47.891530 23973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:47.891971 23967 server_base.cc:1034] running on GCE node
I20260620 07:15:47.892401 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:47.893167 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:47.894368 23967 hybrid_clock.cc:648] HybridClock initialized: now 1781939747894334 us; error 51 us; skew 500 ppm
I20260620 07:15:47.896479 23967 webserver.cc:460] Webserver started at http://127.3.245.2:33997/ using document root <none> and password file <none>
I20260620 07:15:47.897259 23967 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:47.897332 23967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:47.897504 23967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:47.899644 23967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data/instance:
uuid: "77b80758a6044eb696063f7ffc7e9bbf"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "d91be8025c770dd9e37296c25463071a"
server_key_iv: "4d9c93f9ec7639011f4096f1c57d503d"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.900243 23967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance:
uuid: "77b80758a6044eb696063f7ffc7e9bbf"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "d91be8025c770dd9e37296c25463071a"
server_key_iv: "4d9c93f9ec7639011f4096f1c57d503d"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.904175 23967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:47.906857 23981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.908306 23967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:15:47.908455 23967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal
uuid: "77b80758a6044eb696063f7ffc7e9bbf"
format_stamp: "Formatted at 2026-06-20 07:15:47 on dist-test-slave-ldfr"
server_key: "d91be8025c770dd9e37296c25463071a"
server_key_iv: "4d9c93f9ec7639011f4096f1c57d503d"
server_key_version: "encryptionkey@0"
I20260620 07:15:47.908581 23967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:47.930136 23967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:47.930938 23967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:47.931140 23967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:47.931756 23967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:47.932763 23967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:47.932812 23967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.932878 23967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:47.932914 23967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:47.941726 23967 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.2:38649
I20260620 07:15:47.941751 24094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.2:38649 every 8 connection(s)
I20260620 07:15:47.942595 23967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/data/info.pb
I20260620 07:15:47.951032 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 23967
I20260620 07:15:47.951215 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal/instance
I20260620 07:15:47.951581 4052 external_mini_cluster.cc:1428] Setting key f331c228765d27f3c958bce87e492d30
I20260620 07:15:47.951823 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.944402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.2:47265 (local address 127.3.245.62:43295)
0620 07:15:47.944536 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:47.944540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:47.945592 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 07:15:47.946978 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:47.946981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:47.947040 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:47.947150 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:47.948033 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.948569 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.949784 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.950007 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.950114 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.951270 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:47.951277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:47.951281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:47.951296 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:47.951321 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:47.951331 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:47.951417 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:47.951608 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:15:47.951683 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:15:47.952924 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:47.944806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43295 (local address 127.3.245.2:47265)
0620 07:15:47.945440 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:47.945480 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:47.946769 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:47.947333 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:47.947339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:47.947420 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:47.947885 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.947908 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.948773 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.948782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.949648 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:47.949659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:47.950163 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:47.950168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:47.950331 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:47.950979 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:47.950981 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:47.951020 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:47.951083 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:47.951450 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:47.951459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:47.951464 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 07:15:47.951578 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 07:15:47.951969 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260620 07:15:47.953714 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.245.3:0
--local_ip_for_outbound_sockets=127.3.245.3
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--builtin_ntp_servers=127.3.245.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:15:47.954242 24095 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43295
I20260620 07:15:47.954489 24095 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:47.955046 24095 heartbeater.cc:511] Master 127.3.245.62:43295 requested a full tablet report, sending...
I20260620 07:15:47.956249 23776 ts_manager.cc:194] Registered new tserver with Master: 77b80758a6044eb696063f7ffc7e9bbf (127.3.245.2:38649)
I20260620 07:15:47.956955 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.2:47265
W20260620 07:15:48.068507 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:48.068885 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:48.068926 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:48.073089 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:48.073155 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:48.073220 24098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.245.3
I20260620 07:15:48.078007 24098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.245.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.245.3
--webserver_port=0
--tserver_master_addrs=127.3.245.62:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.245.3
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:48.079052 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:48.080446 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:48.090420 24104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:48.090420 24106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:48.090512 24098 server_base.cc:1034] running on GCE node
W20260620 07:15:48.090422 24103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:48.091138 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:48.091820 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:48.092979 24098 hybrid_clock.cc:648] HybridClock initialized: now 1781939748092952 us; error 41 us; skew 500 ppm
I20260620 07:15:48.094844 24098 webserver.cc:460] Webserver started at http://127.3.245.3:37653/ using document root <none> and password file <none>
I20260620 07:15:48.095417 24098 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:48.095479 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:48.095726 24098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:48.098006 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data/instance:
uuid: "d8cade9134b044f4b30a92ff9b00e687"
format_stamp: "Formatted at 2026-06-20 07:15:48 on dist-test-slave-ldfr"
server_key: "927ee59d6dd372213aace1edf59df709"
server_key_iv: "489bc6f51f1ae10a954dac058de7529f"
server_key_version: "encryptionkey@0"
I20260620 07:15:48.098542 24098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance:
uuid: "d8cade9134b044f4b30a92ff9b00e687"
format_stamp: "Formatted at 2026-06-20 07:15:48 on dist-test-slave-ldfr"
server_key: "927ee59d6dd372213aace1edf59df709"
server_key_iv: "489bc6f51f1ae10a954dac058de7529f"
server_key_version: "encryptionkey@0"
I20260620 07:15:48.102295 24098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:15:48.104772 24112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:48.106014 24098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:48.106137 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal
uuid: "d8cade9134b044f4b30a92ff9b00e687"
format_stamp: "Formatted at 2026-06-20 07:15:48 on dist-test-slave-ldfr"
server_key: "927ee59d6dd372213aace1edf59df709"
server_key_iv: "489bc6f51f1ae10a954dac058de7529f"
server_key_version: "encryptionkey@0"
I20260620 07:15:48.106242 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:48.126884 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:48.127576 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:48.127780 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:48.128357 24098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:15:48.129339 24098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:15:48.129384 24098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:48.129457 24098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:15:48.129496 24098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:48.138038 24098 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.3:43791
I20260620 07:15:48.138110 24225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.3:43791 every 8 connection(s)
I20260620 07:15:48.138953 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/data/info.pb
I20260620 07:15:48.141160 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24098
I20260620 07:15:48.141278 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal/instance
I20260620 07:15:48.141618 4052 external_mini_cluster.cc:1428] Setting key b854cfb747f9580b1086cbc7dfb7dd23
I20260620 07:15:48.149432 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.140829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:38187 (local address 127.3.245.62:43295)
0620 07:15:48.140965 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.140969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.142428 (+ 1459us) server_negotiation.cc:407] Connection header received
0620 07:15:48.144206 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.144210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.144267 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.144360 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:48.145394 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.145975 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.147233 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.147483 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.147601 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.148888 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:48.148892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:48.148898 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:48.148915 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:48.148935 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:48.148943 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:48.149033 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.149217 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.149281 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:15:48.150292 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.141276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:43295 (local address 127.3.245.3:38187)
0620 07:15:48.142281 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.142326 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.143992 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.144549 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.144557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.144737 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:15:48.145250 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.145263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.146114 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.146117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.147090 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.147100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.147653 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.147658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.147836 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.148534 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:15:48.148536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:15:48.148577 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:15:48.148713 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:15:48.149034 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:15:48.149047 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:15:48.149055 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.149196 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.149485 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":149,"threads_started":1}
I20260620 07:15:48.151546 24226 heartbeater.cc:347] Connected to a master server at 127.3.245.62:43295
I20260620 07:15:48.151841 24226 heartbeater.cc:464] Registering TS with master...
I20260620 07:15:48.152374 24226 heartbeater.cc:511] Master 127.3.245.62:43295 requested a full tablet report, sending...
I20260620 07:15:48.153391 23776 ts_manager.cc:194] Registered new tserver with Master: d8cade9134b044f4b30a92ff9b00e687 (127.3.245.3:43791)
I20260620 07:15:48.154024 23776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.245.3:38187
I20260620 07:15:48.155581 4052 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:15:48.160905 23815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.157517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.3.245.62:43295)
0620 07:15:48.157633 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.157637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.157798 (+ 161us) server_negotiation.cc:407] Connection header received
0620 07:15:48.157879 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.157881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.157917 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.157975 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:48.158178 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.158676 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.159468 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.159670 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.159744 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.160540 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:48.160542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:48.160544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:48.160555 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:48.160572 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:48.160579 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:48.160702 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.160759 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.160805 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:15:48.163650 23776 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:15:48.166734 23776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:15:48.181427 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.175731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.3:43791 (local address 127.0.0.1:52130)
0620 07:15:48.176382 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.176434 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.176589 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.177200 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.177206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.177227 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:48.177368 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.177375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.178493 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.178499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.179831 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.179841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.180908 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.180912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.181012 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.181047 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.181141 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.181252 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260620 07:15:48.181442 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.175937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.3.245.3:43791)
0620 07:15:48.176351 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.176361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.176753 (+ 392us) server_negotiation.cc:407] Connection header received
0620 07:15:48.176838 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.176845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.177057 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.177213 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:48.177500 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.178394 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.179960 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.180787 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.180933 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.181126 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.181178 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.181278 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":120,"threads_started":1}
I20260620 07:15:48.181901 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.175688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.3.245.2:38649)
0620 07:15:48.176187 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.176200 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.176516 (+ 316us) server_negotiation.cc:407] Connection header received
0620 07:15:48.176843 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.176849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.177055 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.177179 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:48.177589 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.178753 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.180179 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.180776 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.180866 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.181052 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.181380 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.181489 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":160,"threads_started":1}
I20260620 07:15:48.182345 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.176528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:45901 (local address 127.0.0.1:45866)
0620 07:15:48.177476 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.177487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.177591 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.178072 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.178075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.178093 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:48.178184 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.178189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.179333 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.179339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.180832 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.180845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.181885 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.181889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.181984 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.182011 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.182084 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.182189 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":63,"thread_start_us":114,"threads_started":1}
I20260620 07:15:48.182344 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.175546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:38649 (local address 127.0.0.1:58730)
0620 07:15:48.176382 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.176435 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.176593 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.177355 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.177357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.177371 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:48.177470 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.177477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.178871 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.178875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.180045 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.180055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.181148 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.181152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.181226 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.181245 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.182085 (+ 840us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.182189 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":45,"thread_start_us":95,"threads_started":1}
I20260620 07:15:48.184432 24160 tablet_service.cc:1467] Processing CreateTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 (DEFAULT_TABLE table=test-table [id=7220fe0eedda4dae8e93d3afda87ab56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:48.184963 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 (DEFAULT_TABLE table=test-table [id=7220fe0eedda4dae8e93d3afda87ab56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:48.185483 24160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722e2cf8be54da3b66f5e44b0fb01a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:48.185837 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722e2cf8be54da3b66f5e44b0fb01a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:48.191505 24243 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Bootstrap starting.
I20260620 07:15:48.182332 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.176736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.3.245.1:45901)
0620 07:15:48.177054 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.177067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.177684 (+ 617us) server_negotiation.cc:407] Connection header received
0620 07:15:48.177759 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.177764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.177929 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.178070 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:48.178310 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.179217 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.180978 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.181764 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.181872 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.182028 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.182105 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.182185 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260620 07:15:48.193264 24244 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Bootstrap starting.
I20260620 07:15:48.193975 24243 tablet_bootstrap.cc:654] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:48.194849 24243 log.cc:824] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:48.195083 24244 tablet_bootstrap.cc:654] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:48.195207 23898 tablet_service.cc:1467] Processing CreateTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 (DEFAULT_TABLE table=test-table [id=7220fe0eedda4dae8e93d3afda87ab56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:15:48.195799 24244 log.cc:824] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:48.196331 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4722e2cf8be54da3b66f5e44b0fb01a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:48.196956 24243 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: No bootstrap required, opened a new log
I20260620 07:15:48.197152 24243 ts_tablet_manager.cc:1397] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 07:15:48.197679 24244 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: No bootstrap required, opened a new log
I20260620 07:15:48.197881 24244 ts_tablet_manager.cc:1397] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 07:15:48.201025 24244 raft_consensus.cc:348] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.201265 24244 raft_consensus.cc:374] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:48.201323 24244 raft_consensus.cc:729] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b80758a6044eb696063f7ffc7e9bbf, State: Initialized, Role: FOLLOWER
I20260620 07:15:48.201478 24243 raft_consensus.cc:348] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.201733 24243 raft_consensus.cc:374] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:48.201795 24243 raft_consensus.cc:729] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8cade9134b044f4b30a92ff9b00e687, State: Initialized, Role: FOLLOWER
I20260620 07:15:48.201789 24244 consensus_queue.cc:260] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.202270 24243 consensus_queue.cc:260] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.202610 24095 heartbeater.cc:503] Master 127.3.245.62:43295 was elected leader, sending a full tablet report...
I20260620 07:15:48.202873 24244 ts_tablet_manager.cc:1428] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 07:15:48.203692 24226 heartbeater.cc:503] Master 127.3.245.62:43295 was elected leader, sending a full tablet report...
I20260620 07:15:48.203883 24243 ts_tablet_manager.cc:1428] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:15:48.204288 24249 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Bootstrap starting.
I20260620 07:15:48.206626 24249 tablet_bootstrap.cc:654] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:48.207381 24249 log.cc:824] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:48.209302 24249 tablet_bootstrap.cc:492] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: No bootstrap required, opened a new log
I20260620 07:15:48.209496 24249 ts_tablet_manager.cc:1397] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 07:15:48.212874 24249 raft_consensus.cc:348] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.213141 24249 raft_consensus.cc:374] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:48.213203 24249 raft_consensus.cc:729] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c762c4445a7466497710857efa6cc0f, State: Initialized, Role: FOLLOWER
I20260620 07:15:48.213603 24249 consensus_queue.cc:260] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.214288 23964 heartbeater.cc:503] Master 127.3.245.62:43295 was elected leader, sending a full tablet report...
I20260620 07:15:48.214601 24249 ts_tablet_manager.cc:1428] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 07:15:48.249737 23965 tablet.cc:2369] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:15:48.331173 24248 raft_consensus.cc:482] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:15:48.331423 24248 raft_consensus.cc:504] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.332594 24248 leader_election.cc:288] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b80758a6044eb696063f7ffc7e9bbf (127.3.245.2:38649), 4c762c4445a7466497710857efa6cc0f (127.3.245.1:45901)
I20260620 07:15:48.337652 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.332991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:45901 (local address 127.3.245.3:54779)
0620 07:15:48.333096 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.333113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.333209 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.333524 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.333530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.333556 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:48.333702 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.333716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.334750 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.334753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.336151 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.336163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.337056 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.337063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.337240 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.337285 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.337355 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.337475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":110,"spinlock_wait_cycles":8960}
I20260620 07:15:48.337672 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.333314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:50929 (local address 127.3.245.2:38649)
0620 07:15:48.333473 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.333479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.333579 (+ 100us) server_negotiation.cc:407] Connection header received
0620 07:15:48.333703 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.333705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.333747 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.333847 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:48.334088 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.334796 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.336286 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.336903 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.337013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.337077 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.337444 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.337541 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:15:48.337672 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.333102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.245.3:54779 (local address 127.3.245.1:45901)
0620 07:15:48.333247 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.333252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.333269 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:15:48.333328 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.333333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.333392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.333489 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:15:48.333826 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.334613 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.336287 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.336903 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.337013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.337077 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.337444 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.337551 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:15:48.337666 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.332991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.2:38649 (local address 127.3.245.3:50929)
0620 07:15:48.333476 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.333492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:15:48.333606 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:15:48.333870 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:15:48.333872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:15:48.333889 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:15:48.333980 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.333984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.334901 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.334904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.336156 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:15:48.336163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.337056 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.337065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:15:48.337246 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.337269 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:15:48.337355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 07:15:48.337475 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":175,"spinlock_wait_cycles":2688,"thread_start_us":136,"threads_started":1}
I20260620 07:15:48.338474 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722e2cf8be54da3b66f5e44b0fb01a2" candidate_uuid: "d8cade9134b044f4b30a92ff9b00e687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b80758a6044eb696063f7ffc7e9bbf" is_pre_election: true
I20260620 07:15:48.338465 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722e2cf8be54da3b66f5e44b0fb01a2" candidate_uuid: "d8cade9134b044f4b30a92ff9b00e687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c762c4445a7466497710857efa6cc0f" is_pre_election: true
I20260620 07:15:48.338804 23918 raft_consensus.cc:2454] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8cade9134b044f4b30a92ff9b00e687 in term 0.
I20260620 07:15:48.338804 24049 raft_consensus.cc:2454] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8cade9134b044f4b30a92ff9b00e687 in term 0.
I20260620 07:15:48.339320 24115 leader_election.cc:302] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b80758a6044eb696063f7ffc7e9bbf, d8cade9134b044f4b30a92ff9b00e687; no voters:
I20260620 07:15:48.339679 24248 raft_consensus.cc:2781] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:15:48.339777 24248 raft_consensus.cc:482] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:15:48.339834 24248 raft_consensus.cc:3037] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:48.340963 24248 raft_consensus.cc:504] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.341348 24248 leader_election.cc:288] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [CANDIDATE]: Term 1 election: Requested vote from peers 77b80758a6044eb696063f7ffc7e9bbf (127.3.245.2:38649), 4c762c4445a7466497710857efa6cc0f (127.3.245.1:45901)
I20260620 07:15:48.341709 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722e2cf8be54da3b66f5e44b0fb01a2" candidate_uuid: "d8cade9134b044f4b30a92ff9b00e687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b80758a6044eb696063f7ffc7e9bbf"
I20260620 07:15:48.341817 24049 raft_consensus.cc:3037] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:48.341853 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4722e2cf8be54da3b66f5e44b0fb01a2" candidate_uuid: "d8cade9134b044f4b30a92ff9b00e687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c762c4445a7466497710857efa6cc0f"
I20260620 07:15:48.342031 23918 raft_consensus.cc:3037] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:48.342969 24049 raft_consensus.cc:2454] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8cade9134b044f4b30a92ff9b00e687 in term 1.
I20260620 07:15:48.343180 23918 raft_consensus.cc:2454] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8cade9134b044f4b30a92ff9b00e687 in term 1.
I20260620 07:15:48.343365 24115 leader_election.cc:302] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b80758a6044eb696063f7ffc7e9bbf, d8cade9134b044f4b30a92ff9b00e687; no voters:
I20260620 07:15:48.343612 24248 raft_consensus.cc:2781] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:48.343897 24248 raft_consensus.cc:686] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 1 LEADER]: Becoming Leader. State: Replica: d8cade9134b044f4b30a92ff9b00e687, State: Running, Role: LEADER
I20260620 07:15:48.344244 24248 consensus_queue.cc:237] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } }
I20260620 07:15:48.347369 23774 catalog_manager.cc:5495] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8cade9134b044f4b30a92ff9b00e687 (127.3.245.3). New cstate: current_term: 1 leader_uuid: "d8cade9134b044f4b30a92ff9b00e687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8cade9134b044f4b30a92ff9b00e687" member_type: VOTER last_known_addr { host: "127.3.245.3" port: 43791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 } health_report { overall_health: UNKNOWN } } }
I20260620 07:15:48.372846 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.369243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.3.245.3:43791)
0620 07:15:48.369397 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:48.369403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:48.369422 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:15:48.369511 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:48.369514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:48.369576 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:48.369690 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:15:48.369928 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.370444 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.371318 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:48.371574 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:48.371685 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:48.372002 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:15:48.372402 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:15:48.372528 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:48.372636 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:15:48.372687 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:15:48.378648 23918 raft_consensus.cc:1261] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 4c762c4445a7466497710857efa6cc0f [term 1 FOLLOWER]: Refusing update from remote peer d8cade9134b044f4b30a92ff9b00e687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:48.379355 24049 raft_consensus.cc:1261] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 1 FOLLOWER]: Refusing update from remote peer d8cade9134b044f4b30a92ff9b00e687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:15:48.379372 24248 consensus_queue.cc:1035] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c762c4445a7466497710857efa6cc0f" member_type: VOTER last_known_addr { host: "127.3.245.1" port: 45901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:48.379976 24248 consensus_queue.cc:1035] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b80758a6044eb696063f7ffc7e9bbf" member_type: VOTER last_known_addr { host: "127.3.245.2" port: 38649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:15:48.387640 24258 mvcc.cc:205] Tried to move back new op lower bound from 7298825209351413760 to 7298825209220939776. Current Snapshot: MvccSnapshot[applied={T|T < 7298825209351413760}]
I20260620 07:15:48.390830 24257 mvcc.cc:205] Tried to move back new op lower bound from 7298825209351413760 to 7298825209220939776. Current Snapshot: MvccSnapshot[applied={T|T < 7298825209351413760 or (T in {7298825209351413760})}]
I20260620 07:15:48.395218 23774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:48.395394 23774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:15:48.398056 23774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b60c7c67d5e4c2a8aa84d11082f9d0c: Sending DeleteTablet for 3 replicas of tablet 4722e2cf8be54da3b66f5e44b0fb01a2
I20260620 07:15:48.398865 24160 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:48 UTC) from {username='slave'} at 127.0.0.1:52130
I20260620 07:15:48.398866 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:48 UTC) from {username='slave'} at 127.0.0.1:58730
I20260620 07:15:48.399096 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet 4722e2cf8be54da3b66f5e44b0fb01a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:15:48 UTC) from {username='slave'} at 127.0.0.1:45866
I20260620 07:15:48.399500 24266 tablet_replica.cc:330] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: stopping tablet replica
I20260620 07:15:48.399861 24266 raft_consensus.cc:2229] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:15:48.399979 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23836
I20260620 07:15:48.400291 24266 raft_consensus.cc:2258] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:48.401000 24265 tablet_replica.cc:330] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: stopping tablet replica
I20260620 07:15:48.401315 24265 raft_consensus.cc:2229] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:15:48.401624 24265 raft_consensus.cc:2258] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:15:48.403200 24266 ts_tablet_manager.cc:1905] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:48.403206 24265 ts_tablet_manager.cc:1905] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:15:48.406049 24265 ts_tablet_manager.cc:1918] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:48.406153 24265 log.cc:1196] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-1/wal/wals/4722e2cf8be54da3b66f5e44b0fb01a2
I20260620 07:15:48.406467 24265 ts_tablet_manager.cc:1939] T 4722e2cf8be54da3b66f5e44b0fb01a2 P 77b80758a6044eb696063f7ffc7e9bbf: Deleting consensus metadata
I20260620 07:15:48.406697 24266 ts_tablet_manager.cc:1918] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:15:48.406795 24266 log.cc:1196] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Deleting WAL directory at /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781939488540940-4052-0/minicluster-data/ts-2/wal/wals/4722e2cf8be54da3b66f5e44b0fb01a2
I20260620 07:15:48.407117 24266 ts_tablet_manager.cc:1939] T 4722e2cf8be54da3b66f5e44b0fb01a2 P d8cade9134b044f4b30a92ff9b00e687: Deleting consensus metadata
I20260620 07:15:48.407753 23762 catalog_manager.cc:4841] TS 77b80758a6044eb696063f7ffc7e9bbf (127.3.245.2:38649): tablet 4722e2cf8be54da3b66f5e44b0fb01a2 (table test-table [id=7220fe0eedda4dae8e93d3afda87ab56]) successfully deleted
I20260620 07:15:48.408294 23760 catalog_manager.cc:4841] TS d8cade9134b044f4b30a92ff9b00e687 (127.3.245.3:43791): tablet 4722e2cf8be54da3b66f5e44b0fb01a2 (table test-table [id=7220fe0eedda4dae8e93d3afda87ab56]) successfully deleted
W20260620 07:15:48.410512 23763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.245.1:45901 (error 108)
I20260620 07:15:48.411409 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23967
I20260620 07:15:48.411682 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:48.411336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.1:45901 (local address 127.0.0.1:45876)
0620 07:15:48.411491 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:15:48.411576 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.1:45901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:15:48.413343 23763 catalog_manager.cc:4584] TS 4c762c4445a7466497710857efa6cc0f (127.3.245.1:45901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4722e2cf8be54da3b66f5e44b0fb01a2: Network error: Client connection negotiation failed: client connection to 127.3.245.1:45901: connect: Connection refused (error 111)
I20260620 07:15:48.419011 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24098
I20260620 07:15:48.424880 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 23745
2026-06-20T07:15:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1150 ms)
[----------] 6 tests from EncryptionPolicyTest (11174 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:48 dist-test-slave-ldfr krb5kdc[24271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:48 dist-test-slave-ldfr krb5kdc[24271](info): set up 2 sockets
Jun 20 07:15:48 dist-test-slave-ldfr krb5kdc[24271](info): commencing operation
krb5kdc: starting...
W20260620 07:15:49.492908 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:49.616377 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:49.726390 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:49.726665 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:49.726707 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:49.730756 24284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:49.730824 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:49.730844 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:49.730861 24284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:49.730877 24284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:49.736025 24284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:49.737118 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:49.738091 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:49.746678 24290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:49.746696 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:49.746696 24289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:49.746982 24284 server_base.cc:1034] running on GCE node
I20260620 07:15:49.747601 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:49.748759 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:49.749923 24284 hybrid_clock.cc:648] HybridClock initialized: now 1781939749749910 us; error 26 us; skew 500 ppm
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:49.752692 24284 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:49.753561 24284 webserver.cc:460] Webserver started at http://127.0.0.1:45669/ using document root <none> and password file <none>
I20260620 07:15:49.754077 24284 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:49.754136 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:49.754304 24284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:49.756297 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "5e063db22a0b49c191bd3d35985edb60"
format_stamp: "Formatted at 2026-06-20 07:15:49 on dist-test-slave-ldfr"
server_key: "d5ca86e74c07de832a79fda1b67cc661"
server_key_iv: "58a2401445a5dfd0affa9a2dc8fab28d"
server_key_version: "encryptionkey@0"
I20260620 07:15:49.756880 24284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "5e063db22a0b49c191bd3d35985edb60"
format_stamp: "Formatted at 2026-06-20 07:15:49 on dist-test-slave-ldfr"
server_key: "d5ca86e74c07de832a79fda1b67cc661"
server_key_iv: "58a2401445a5dfd0affa9a2dc8fab28d"
server_key_version: "encryptionkey@0"
I20260620 07:15:49.760423 24284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:49.762890 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:49.764169 24284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:15:49.764305 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "5e063db22a0b49c191bd3d35985edb60"
format_stamp: "Formatted at 2026-06-20 07:15:49 on dist-test-slave-ldfr"
server_key: "d5ca86e74c07de832a79fda1b67cc661"
server_key_iv: "58a2401445a5dfd0affa9a2dc8fab28d"
server_key_version: "encryptionkey@0"
I20260620 07:15:49.764418 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:49.809428 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:49.812492 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:49.812753 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:49.819793 24284 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41893
I20260620 07:15:49.819823 24351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41893 every 8 connection(s)
I20260620 07:15:49.820784 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:49.823194 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24284
I20260620 07:15:49.823344 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:49.823661 4052 external_mini_cluster.cc:1428] Setting key ffe0accd662df4a90053d78b9c56ec4b
I20260620 07:15:49.823714 24352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:49.829952 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60: Bootstrap starting.
I20260620 07:15:49.832315 24352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60: Neither blocks nor log segments found. Creating new log.
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:15:49.833124 24352 log.cc:824] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:49.835489 24352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60: No bootstrap required, opened a new log
I20260620 07:15:49.838624 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:49.825138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42077 (local address 127.0.0.1:41893)
0620 07:15:49.825590 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:49.825604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:49.825638 (+ 34us) server_negotiation.cc:407] Connection header received
0620 07:15:49.826479 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:49.826497 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:49.826904 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:49.827261 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:49.828073 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:49.829134 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:49.829971 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:49.830478 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:49.830653 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:49.833370 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:49.833375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:49.833390 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:49.833425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:49.835374 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:49.835931 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:49.835937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:49.835942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:49.836080 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:49.836419 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:49.836423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:49.836425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:49.836938 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:49.837144 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:49.837366 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 07:15:49.837591 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":151,"threads_started":1}
I20260620 07:15:49.840144 24352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } }
I20260620 07:15:49.840399 24352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:49.840464 24352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e063db22a0b49c191bd3d35985edb60, State: Initialized, Role: FOLLOWER
I20260620 07:15:49.840914 24352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } }
I20260620 07:15:49.841091 24352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:49.841173 24352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:49.841288 24352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:49.841676 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:49.842283 24352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } }
I20260620 07:15:49.842619 24352 leader_election.cc:302] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e063db22a0b49c191bd3d35985edb60; no voters:
I20260620 07:15:49.842932 24352 leader_election.cc:288] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:49.843081 24359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:49.843322 24359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [term 1 LEADER]: Becoming Leader. State: Replica: 5e063db22a0b49c191bd3d35985edb60, State: Running, Role: LEADER
I20260620 07:15:49.843641 24359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } }
I20260620 07:15:49.844214 24352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:49.844894 24363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e063db22a0b49c191bd3d35985edb60. Latest consensus state: current_term: 1 leader_uuid: "5e063db22a0b49c191bd3d35985edb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } } }
I20260620 07:15:49.845114 24363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:49.845400 24362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e063db22a0b49c191bd3d35985edb60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e063db22a0b49c191bd3d35985edb60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41893 } } }
I20260620 07:15:49.845516 24362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:49.845943 24371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:49.849370 24371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:49.853675 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:49.843818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33097 (local address 127.0.0.1:41893)
0620 07:15:49.843950 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:49.843954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:49.844328 (+ 374us) server_negotiation.cc:407] Connection header received
0620 07:15:49.844493 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:49.844498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:49.844552 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:49.844682 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:49.845533 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:49.846399 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:49.847540 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:49.847925 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:49.848024 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:49.849686 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:49.849692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:49.849696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:49.849719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:49.851273 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:49.851860 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:49.851864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:49.851866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:49.851949 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:49.852331 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:49.852334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:49.852336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:49.852540 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:49.853337 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:49.853430 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 07:15:49.853526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
I20260620 07:15:49.855808 24371 catalog_manager.cc:1347] Generated new cluster ID: f96b94ec98834ae7938dee2f3926cbf5
I20260620 07:15:49.855880 24371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:49.869287 24371 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:49.870266 24371 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:49.876312 24371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e063db22a0b49c191bd3d35985edb60: Generated new TSK 0
I20260620 07:15:49.877131 24371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:49.888623 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24284
W20260620 07:15:49.894433 24269 connection.cc:535] client connection to 127.0.0.1:41893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41893): Connection reset by peer (error 104)
2026-06-20T07:15:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24383](info): set up 2 sockets
Jun 20 07:15:49 dist-test-slave-ldfr krb5kdc[24383](info): commencing operation
krb5kdc: starting...
W20260620 07:15:50.928887 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:51 dist-test-slave-ldfr krb5kdc[24383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:51.049815 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:51.160668 24396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:51.161023 24396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:51.161110 24396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:51.165500 24396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:51.165601 24396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:51.165639 24396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:51.165679 24396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:51.165720 24396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:51.171221 24396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:51.172446 24396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:51.173621 24396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:51.184129 24404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:51.184206 24401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:51.184188 24402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:51.184314 24396 server_base.cc:1034] running on GCE node
I20260620 07:15:51.184918 24396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:51.186076 24396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:51.187269 24396 hybrid_clock.cc:648] HybridClock initialized: now 1781939751187256 us; error 25 us; skew 500 ppm
Jun 20 07:15:51 dist-test-slave-ldfr krb5kdc[24383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:51.190044 24396 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:51.190896 24396 webserver.cc:460] Webserver started at http://127.0.0.1:34519/ using document root <none> and password file <none>
I20260620 07:15:51.191406 24396 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:51.191462 24396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:51.191627 24396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:51.193672 24396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "9b836275f64a4c6984fe482ebd1946d7"
format_stamp: "Formatted at 2026-06-20 07:15:51 on dist-test-slave-ldfr"
server_key: "5bae7825fe25b772ca333ca2fc4f15db"
server_key_iv: "b63f61702b2515d776cc774cc00d7cae"
server_key_version: "encryptionkey@0"
I20260620 07:15:51.194170 24396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "9b836275f64a4c6984fe482ebd1946d7"
format_stamp: "Formatted at 2026-06-20 07:15:51 on dist-test-slave-ldfr"
server_key: "5bae7825fe25b772ca333ca2fc4f15db"
server_key_iv: "b63f61702b2515d776cc774cc00d7cae"
server_key_version: "encryptionkey@0"
I20260620 07:15:51.197868 24396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:15:51.200212 24411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:51.201484 24396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:15:51.201586 24396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "9b836275f64a4c6984fe482ebd1946d7"
format_stamp: "Formatted at 2026-06-20 07:15:51 on dist-test-slave-ldfr"
server_key: "5bae7825fe25b772ca333ca2fc4f15db"
server_key_iv: "b63f61702b2515d776cc774cc00d7cae"
server_key_version: "encryptionkey@0"
I20260620 07:15:51.201713 24396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:51.215627 24396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:51.218559 24396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:51.218772 24396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:51.225896 24396 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46555
I20260620 07:15:51.225922 24463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46555 every 8 connection(s)
I20260620 07:15:51.226850 24396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:51.229559 24464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:51.234921 24464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7: Bootstrap starting.
I20260620 07:15:51.236311 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24396
I20260620 07:15:51.236459 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:51.236770 4052 external_mini_cluster.cc:1428] Setting key 7184520fd40f9d58e0191688d6653ff1
I20260620 07:15:51.237668 24464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:51.238544 24464 log.cc:824] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:51.240787 24464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7: No bootstrap required, opened a new log
I20260620 07:15:51.245056 24464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } }
I20260620 07:15:51.245293 24464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:51.245368 24464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b836275f64a4c6984fe482ebd1946d7, State: Initialized, Role: FOLLOWER
I20260620 07:15:51.245920 24464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } }
I20260620 07:15:51.246081 24464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:51.246181 24464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:51.246292 24464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:51.246843 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:51.237929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36337 (local address 127.0.0.1:46555)
0620 07:15:51.238407 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:51.238428 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:51.238473 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:15:51.239267 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:51.239318 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:51.239735 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:51.240108 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:51.240518 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:51.241977 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:51.242879 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:51.243704 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:51.243902 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:51.244843 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:51.244848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:51.244862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:51.245295 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:51.245377 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:51.245409 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:51.245553 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:51.245750 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 07:15:51.245895 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":129,"threads_started":1}
I20260620 07:15:51.247344 24464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } }
I20260620 07:15:51.247694 24464 leader_election.cc:302] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b836275f64a4c6984fe482ebd1946d7; no voters:
I20260620 07:15:51.248073 24464 leader_election.cc:288] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:51.248147 24469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:51.248370 24469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [term 1 LEADER]: Becoming Leader. State: Replica: 9b836275f64a4c6984fe482ebd1946d7, State: Running, Role: LEADER
I20260620 07:15:51.248689 24469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } }
I20260620 07:15:51.249449 24464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:51.250232 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:51.250484 24470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b836275f64a4c6984fe482ebd1946d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } } }
I20260620 07:15:51.250651 24470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:51.250916 24471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b836275f64a4c6984fe482ebd1946d7. Latest consensus state: current_term: 1 leader_uuid: "9b836275f64a4c6984fe482ebd1946d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b836275f64a4c6984fe482ebd1946d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46555 } } }
I20260620 07:15:51.251003 24471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:51.251137 24478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:51.254663 24478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:51.258563 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:51.252581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36835 (local address 127.0.0.1:46555)
0620 07:15:51.252719 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:51.252724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:51.253662 (+ 938us) server_negotiation.cc:407] Connection header received
0620 07:15:51.253801 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:51.253804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:51.253872 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:51.253955 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:51.254315 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:51.255070 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:51.256285 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:51.256670 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:51.256774 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:51.257882 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:51.257887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:51.257890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:51.257903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:51.257931 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:51.257945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:51.258023 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:51.258368 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 07:15:51.258408 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:15:51.260748 24478 catalog_manager.cc:1347] Generated new cluster ID: 3958f98599954f73ae237afbb4b8dc40
I20260620 07:15:51.260854 24478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:51.272115 24478 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:51.273375 24478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:51.286060 24478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b836275f64a4c6984fe482ebd1946d7: Generated new TSK 0
I20260620 07:15:51.287168 24478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:51.296149 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24396
W20260620 07:15:51.301815 24381 connection.cc:535] client connection to 127.0.0.1:46555 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46555): Connection reset by peer (error 104)
2026-06-20T07:15:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:51 dist-test-slave-ldfr krb5kdc[24495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:51 dist-test-slave-ldfr krb5kdc[24495](info): set up 2 sockets
Jun 20 07:15:51 dist-test-slave-ldfr krb5kdc[24495](info): commencing operation
krb5kdc: starting...
W20260620 07:15:52.340943 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:52 dist-test-slave-ldfr krb5kdc[24495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:52.464298 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:52.576000 24508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:52.576347 24508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:52.576434 24508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:52.580554 24508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:52.580677 24508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:52.580726 24508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:52.580789 24508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:52.580827 24508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:52.586491 24508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:52.587621 24508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:52.588672 24508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:52.597154 24514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:52.597329 24508 server_base.cc:1034] running on GCE node
W20260620 07:15:52.597175 24516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:52.597160 24513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:52.598028 24508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:52.599125 24508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:52.600311 24508 hybrid_clock.cc:648] HybridClock initialized: now 1781939752600297 us; error 25 us; skew 500 ppm
Jun 20 07:15:52 dist-test-slave-ldfr krb5kdc[24495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:52.602968 24508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:52.603853 24508 webserver.cc:460] Webserver started at http://127.0.0.1:43105/ using document root <none> and password file <none>
I20260620 07:15:52.604420 24508 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:52.604513 24508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:52.604797 24508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:52.606865 24508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "cfae062c7d5246afb507b0d88689ca1f"
format_stamp: "Formatted at 2026-06-20 07:15:52 on dist-test-slave-ldfr"
server_key: "512feaad4fe9f80a2dd827bfd9d0f560"
server_key_iv: "2a6ad634b86da43bdbaa970c7b53c409"
server_key_version: "encryptionkey@0"
I20260620 07:15:52.607436 24508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "cfae062c7d5246afb507b0d88689ca1f"
format_stamp: "Formatted at 2026-06-20 07:15:52 on dist-test-slave-ldfr"
server_key: "512feaad4fe9f80a2dd827bfd9d0f560"
server_key_iv: "2a6ad634b86da43bdbaa970c7b53c409"
server_key_version: "encryptionkey@0"
I20260620 07:15:52.611307 24508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:52.613770 24523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:52.614964 24508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:15:52.615118 24508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "cfae062c7d5246afb507b0d88689ca1f"
format_stamp: "Formatted at 2026-06-20 07:15:52 on dist-test-slave-ldfr"
server_key: "512feaad4fe9f80a2dd827bfd9d0f560"
server_key_iv: "2a6ad634b86da43bdbaa970c7b53c409"
server_key_version: "encryptionkey@0"
I20260620 07:15:52.615239 24508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:52.619935 24508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:52.622786 24508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:52.622991 24508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:52.629700 24508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44879
I20260620 07:15:52.629755 24575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44879 every 8 connection(s)
I20260620 07:15:52.630676 24508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:52.633574 24576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:52.638952 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: Bootstrap starting.
I20260620 07:15:52.640867 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24508
I20260620 07:15:52.641011 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:52.641346 4052 external_mini_cluster.cc:1428] Setting key 7b05c08765c3d22007f20d95f3fadf4a
I20260620 07:15:52.641633 24576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:52.642542 24576 log.cc:824] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:52.644539 24576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: No bootstrap required, opened a new log
I20260620 07:15:52.646399 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:52.642513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44585 (local address 127.0.0.1:44879)
0620 07:15:52.643022 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:52.643040 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:52.643084 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:15:52.643863 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:52.643882 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:52.644158 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:52.644496 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:52.645256 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:52.645262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:52.645280 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:52.645289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:52.645339 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:52.645358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:52.645472 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:52.645581 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 07:15:52.645745 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":136,"threads_started":1}
I20260620 07:15:52.648600 24576 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260620 07:15:52.648910 24576 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:52.648986 24576 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfae062c7d5246afb507b0d88689ca1f, State: Initialized, Role: FOLLOWER
I20260620 07:15:52.649497 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:52.649462 24576 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260620 07:15:52.649613 24576 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:52.649672 24576 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:52.649762 24576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:52.650939 24576 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260620 07:15:52.651304 24576 leader_election.cc:302] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfae062c7d5246afb507b0d88689ca1f; no voters:
I20260620 07:15:52.651641 24576 leader_election.cc:288] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:52.651743 24585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:52.651998 24585 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [term 1 LEADER]: Becoming Leader. State: Replica: cfae062c7d5246afb507b0d88689ca1f, State: Running, Role: LEADER
I20260620 07:15:52.652328 24585 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } }
I20260620 07:15:52.653101 24576 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:52.653688 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfae062c7d5246afb507b0d88689ca1f. Latest consensus state: current_term: 1 leader_uuid: "cfae062c7d5246afb507b0d88689ca1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } } }
I20260620 07:15:52.653877 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:52.654039 24579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:52.651772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43891 (local address 127.0.0.1:44879)
0620 07:15:52.651886 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:52.651890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:52.652153 (+ 263us) server_negotiation.cc:407] Connection header received
0620 07:15:52.652342 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:52.652345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:52.652377 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:52.652425 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:52.653322 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:52.653326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:52.653328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:52.653340 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:52.653369 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:52.653383 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:52.653478 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:52.653798 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 07:15:52.653839 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 07:15:52.654142 24587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfae062c7d5246afb507b0d88689ca1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfae062c7d5246afb507b0d88689ca1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44879 } } }
I20260620 07:15:52.654224 24587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f [sys.catalog]: This master's current role is: LEADER
W20260620 07:15:52.658015 24601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:52.658128 24601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:52.658226 24594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:52.659096 24594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:52.664494 24594 catalog_manager.cc:1347] Generated new cluster ID: eadadb68bd6743658bf4947a37d69729
I20260620 07:15:52.664564 24594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:52.702497 24594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:52.702908 24594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:52.711371 24594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfae062c7d5246afb507b0d88689ca1f: Generated new TSK 0
I20260620 07:15:52.712296 24594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:52.726682 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24508
2026-06-20T07:15:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:52 dist-test-slave-ldfr krb5kdc[24607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:52 dist-test-slave-ldfr krb5kdc[24607](info): set up 2 sockets
Jun 20 07:15:52 dist-test-slave-ldfr krb5kdc[24607](info): commencing operation
krb5kdc: starting...
W20260620 07:15:53.792845 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:53 dist-test-slave-ldfr krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:53.915895 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:54.028229 24620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:54.028551 24620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:54.028671 24620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:54.032658 24620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:54.032754 24620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:54.032794 24620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:54.032830 24620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:54.032862 24620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:54.038208 24620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:54.039358 24620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:54.040400 24620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:54.048995 24628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:54.049208 24620 server_base.cc:1034] running on GCE node
W20260620 07:15:54.049010 24625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:54.049012 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:54.049881 24620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:54.051074 24620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:54.052273 24620 hybrid_clock.cc:648] HybridClock initialized: now 1781939754052253 us; error 28 us; skew 500 ppm
Jun 20 07:15:54 dist-test-slave-ldfr krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:54.055150 24620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:54.056059 24620 webserver.cc:460] Webserver started at http://127.0.0.1:36879/ using document root <none> and password file <none>
I20260620 07:15:54.056694 24620 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:54.056783 24620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:54.056990 24620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:54.059105 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "173a531f848c49068a8716a106eb3b53"
format_stamp: "Formatted at 2026-06-20 07:15:54 on dist-test-slave-ldfr"
server_key: "d3f0290fab2161570a33c6a64a2ed904"
server_key_iv: "33f482555565ad7cda9fb882f1f416ef"
server_key_version: "encryptionkey@0"
I20260620 07:15:54.059693 24620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "173a531f848c49068a8716a106eb3b53"
format_stamp: "Formatted at 2026-06-20 07:15:54 on dist-test-slave-ldfr"
server_key: "d3f0290fab2161570a33c6a64a2ed904"
server_key_iv: "33f482555565ad7cda9fb882f1f416ef"
server_key_version: "encryptionkey@0"
I20260620 07:15:54.063706 24620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:15:54.066303 24635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:54.067726 24620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:15:54.067845 24620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "173a531f848c49068a8716a106eb3b53"
format_stamp: "Formatted at 2026-06-20 07:15:54 on dist-test-slave-ldfr"
server_key: "d3f0290fab2161570a33c6a64a2ed904"
server_key_iv: "33f482555565ad7cda9fb882f1f416ef"
server_key_version: "encryptionkey@0"
I20260620 07:15:54.067993 24620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:54.084585 24620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:54.087751 24620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:54.088018 24620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:54.095094 24620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46203
I20260620 07:15:54.095157 24687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46203 every 8 connection(s)
I20260620 07:15:54.096110 24620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:54.098933 24688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:54.102458 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24620
I20260620 07:15:54.102576 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:54.102876 4052 external_mini_cluster.cc:1428] Setting key f9da0325810b4b7d2019ec8c6004f32e
I20260620 07:15:54.104791 24688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: Bootstrap starting.
I20260620 07:15:54.106988 24688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:54.107738 24688 log.cc:824] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:54.109674 24688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: No bootstrap required, opened a new log
Jun 20 07:15:54 dist-test-slave-ldfr krb5kdc[24607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:15:54.114215 24688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } }
I20260620 07:15:54.114475 24688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:54.114549 24688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173a531f848c49068a8716a106eb3b53, State: Initialized, Role: FOLLOWER
I20260620 07:15:54.115068 24688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } }
I20260620 07:15:54.115273 24688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:54.115399 24688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:54.115510 24688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:54.116739 24688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } }
I20260620 07:15:54.117084 24688 leader_election.cc:302] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173a531f848c49068a8716a106eb3b53; no voters:
I20260620 07:15:54.117427 24688 leader_election.cc:288] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:54.117542 24693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:54.117633 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:54.104302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45959 (local address 127.0.0.1:46203)
0620 07:15:54.104785 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:54.104799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:54.104835 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:15:54.105509 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:54.105543 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:54.105885 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:54.106234 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:54.107060 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:54.108166 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:54.109095 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:54.109575 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:54.109752 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:54.112513 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:54.112518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:54.112532 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:54.112559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:54.114346 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:54.114893 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:54.114901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:54.114906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:54.115030 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:54.115334 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:54.115337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:54.115342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:54.115854 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:54.116004 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:54.116317 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 07:15:54.116663 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":140,"threads_started":1}
I20260620 07:15:54.117767 24693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [term 1 LEADER]: Becoming Leader. State: Replica: 173a531f848c49068a8716a106eb3b53, State: Running, Role: LEADER
I20260620 07:15:54.118275 24693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } }
I20260620 07:15:54.118767 24688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:54.120034 24695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173a531f848c49068a8716a106eb3b53. Latest consensus state: current_term: 1 leader_uuid: "173a531f848c49068a8716a106eb3b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } } }
I20260620 07:15:54.120220 24695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:54.120666 24694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173a531f848c49068a8716a106eb3b53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173a531f848c49068a8716a106eb3b53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46203 } } }
I20260620 07:15:54.120818 24694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:54.121104 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 07:15:54.123806 24708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:54.123927 24708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:54.124034 24705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:54.124814 24705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:54.131106 24705 catalog_manager.cc:1347] Generated new cluster ID: e0f99067f68d4db085b39dc384c3929e
I20260620 07:15:54.131170 24705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:54.132386 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:54.123534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38575 (local address 127.0.0.1:46203)
0620 07:15:54.123634 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:54.123639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:54.123653 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:15:54.123689 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:54.123692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:54.123746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:54.123831 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:54.124673 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:54.125446 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:54.126642 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:54.126980 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:54.127091 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:54.128794 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:54.128798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:54.128800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:54.128826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:54.130553 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:54.131073 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:54.131079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:54.131081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:54.131169 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:54.131486 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:54.131490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:54.131491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:54.131718 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:54.131851 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:54.132121 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 07:15:54.132240 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:15:54.144677 24705 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:54.145673 24705 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:54.153738 24705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173a531f848c49068a8716a106eb3b53: Generated new TSK 0
I20260620 07:15:54.154452 24705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:54.160246 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24620
2026-06-20T07:15:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:54 dist-test-slave-ldfr krb5kdc[24719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:54 dist-test-slave-ldfr krb5kdc[24719](info): set up 2 sockets
Jun 20 07:15:54 dist-test-slave-ldfr krb5kdc[24719](info): commencing operation
krb5kdc: starting...
W20260620 07:15:55.200974 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:55 dist-test-slave-ldfr krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:55.328516 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:55.446820 24732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:55.447099 24732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:55.447141 24732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:55.451257 24732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:55.451327 24732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:55.451347 24732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:55.451364 24732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:55.451417 24732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:55.456813 24732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:55.457897 24732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:55.458935 24732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:55.467720 24737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:55.467715 24740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:55.467726 24732 server_base.cc:1034] running on GCE node
W20260620 07:15:55.467747 24738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:55.468575 24732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:55.469794 24732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:55.470976 24732 hybrid_clock.cc:648] HybridClock initialized: now 1781939755470955 us; error 42 us; skew 500 ppm
Jun 20 07:15:55 dist-test-slave-ldfr krb5kdc[24719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:55.473987 24732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:55.474972 24732 webserver.cc:460] Webserver started at http://127.0.0.1:46503/ using document root <none> and password file <none>
I20260620 07:15:55.475548 24732 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:55.475611 24732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:55.475796 24732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:55.478123 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "120361eca27a4908abb61f210f7f366e"
format_stamp: "Formatted at 2026-06-20 07:15:55 on dist-test-slave-ldfr"
server_key: "b72b751f0c03646585488384f50a39cb"
server_key_iv: "d1d9aebe41e1b540474dbc9c71491107"
server_key_version: "encryptionkey@0"
I20260620 07:15:55.478713 24732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "120361eca27a4908abb61f210f7f366e"
format_stamp: "Formatted at 2026-06-20 07:15:55 on dist-test-slave-ldfr"
server_key: "b72b751f0c03646585488384f50a39cb"
server_key_iv: "d1d9aebe41e1b540474dbc9c71491107"
server_key_version: "encryptionkey@0"
I20260620 07:15:55.482935 24732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:15:55.485818 24747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:55.487222 24732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:55.487483 24732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "120361eca27a4908abb61f210f7f366e"
format_stamp: "Formatted at 2026-06-20 07:15:55 on dist-test-slave-ldfr"
server_key: "b72b751f0c03646585488384f50a39cb"
server_key_iv: "d1d9aebe41e1b540474dbc9c71491107"
server_key_version: "encryptionkey@0"
I20260620 07:15:55.487612 24732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:55.531212 24732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:55.534405 24732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:55.534639 24732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:55.542291 24732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39319
I20260620 07:15:55.542384 24799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39319 every 8 connection(s)
I20260620 07:15:55.544004 24732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:55.546053 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24732
I20260620 07:15:55.546211 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:55.546579 4052 external_mini_cluster.cc:1428] Setting key 9d015f3526294e4faf62a9aedf2013e1
I20260620 07:15:55.547878 24800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:55.554768 24800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e: Bootstrap starting.
I20260620 07:15:55.557716 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:55.548285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58313 (local address 127.0.0.1:39319)
0620 07:15:55.548819 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:55.548844 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:55.548893 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:15:55.549905 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:55.549924 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:55.550312 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:55.550661 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:55.551093 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:55.552085 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:55.553516 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:55.554295 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:55.554459 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:55.555562 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:55.555569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:55.555586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:55.556112 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:55.556187 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:55.556223 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:55.556339 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:55.556489 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 07:15:55.556741 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":155,"threads_started":1}
I20260620 07:15:55.557718 24800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:55.558679 24800 log.cc:824] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:55.560926 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:55.561086 24800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e: No bootstrap required, opened a new log
I20260620 07:15:55.565852 24800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260620 07:15:55.566175 24800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:55.566246 24800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120361eca27a4908abb61f210f7f366e, State: Initialized, Role: FOLLOWER
I20260620 07:15:55.566747 24800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260620 07:15:55.566900 24800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:55.566965 24800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:55.567073 24800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:55.567535 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:55.563091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40165 (local address 127.0.0.1:39319)
0620 07:15:55.563212 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:55.563217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:55.563396 (+ 179us) server_negotiation.cc:407] Connection header received
0620 07:15:55.563476 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:55.563478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:55.563527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:55.563616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:55.563909 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:55.564813 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:55.565611 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:55.565971 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:55.566075 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:55.567063 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:55.567067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:55.567070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:55.567082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:55.567115 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:55.567126 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:55.567206 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:55.567344 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 07:15:55.567398 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:15:55.568279 24800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260620 07:15:55.568686 24800 leader_election.cc:302] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 120361eca27a4908abb61f210f7f366e; no voters:
I20260620 07:15:55.569028 24800 leader_election.cc:288] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:55.569139 24810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:55.569416 24810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [term 1 LEADER]: Becoming Leader. State: Replica: 120361eca27a4908abb61f210f7f366e, State: Running, Role: LEADER
I20260620 07:15:55.569725 24810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } }
I20260620 07:15:55.570299 24800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:55.571398 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 120361eca27a4908abb61f210f7f366e. Latest consensus state: current_term: 1 leader_uuid: "120361eca27a4908abb61f210f7f366e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } } }
I20260620 07:15:55.571528 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:55.571897 24819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:55.571868 24811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "120361eca27a4908abb61f210f7f366e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "120361eca27a4908abb61f210f7f366e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39319 } } }
I20260620 07:15:55.572554 24811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:55.575124 24819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:55.581228 24819 catalog_manager.cc:1347] Generated new cluster ID: ad0efa774deb4a758c877986244ee387
I20260620 07:15:55.581316 24819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:55.593515 24819 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:55.594514 24819 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:55.601774 24819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 120361eca27a4908abb61f210f7f366e: Generated new TSK 0
I20260620 07:15:55.602638 24819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:55.613296 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24732
2026-06-20T07:15:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:55 dist-test-slave-ldfr krb5kdc[24831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:55 dist-test-slave-ldfr krb5kdc[24831](info): set up 2 sockets
Jun 20 07:15:55 dist-test-slave-ldfr krb5kdc[24831](info): commencing operation
krb5kdc: starting...
W20260620 07:15:56.653017 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:56 dist-test-slave-ldfr krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:56.774065 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:56.887849 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:56.888182 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:56.888260 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:56.892458 24844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:56.892570 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:56.892608 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:56.892664 24844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:56.892729 24844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:56.898200 24844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:56.899374 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:56.900451 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:56.909440 24850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:56.909456 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:56.909435 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:56.909691 24844 server_base.cc:1034] running on GCE node
I20260620 07:15:56.910338 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:56.911559 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:56.912745 24844 hybrid_clock.cc:648] HybridClock initialized: now 1781939756912727 us; error 34 us; skew 500 ppm
Jun 20 07:15:56 dist-test-slave-ldfr krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:56.915555 24844 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:56.916507 24844 webserver.cc:460] Webserver started at http://127.0.0.1:33029/ using document root <none> and password file <none>
I20260620 07:15:56.917081 24844 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:56.917145 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:56.917315 24844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:56.919416 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "4fb4fc173c4f45de884c2b2896edbba2"
format_stamp: "Formatted at 2026-06-20 07:15:56 on dist-test-slave-ldfr"
server_key: "6c645eeb10f502d1aad4223d0af803d3"
server_key_iv: "5eb102d7f0909abe8cf800bc07b518ba"
server_key_version: "encryptionkey@0"
I20260620 07:15:56.919930 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "4fb4fc173c4f45de884c2b2896edbba2"
format_stamp: "Formatted at 2026-06-20 07:15:56 on dist-test-slave-ldfr"
server_key: "6c645eeb10f502d1aad4223d0af803d3"
server_key_iv: "5eb102d7f0909abe8cf800bc07b518ba"
server_key_version: "encryptionkey@0"
I20260620 07:15:56.923623 24844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:15:56.926038 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:56.927459 24844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:15:56.927568 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "4fb4fc173c4f45de884c2b2896edbba2"
format_stamp: "Formatted at 2026-06-20 07:15:56 on dist-test-slave-ldfr"
server_key: "6c645eeb10f502d1aad4223d0af803d3"
server_key_iv: "5eb102d7f0909abe8cf800bc07b518ba"
server_key_version: "encryptionkey@0"
I20260620 07:15:56.927722 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:56.932343 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:56.935635 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:56.935859 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:56.943128 24844 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36503
I20260620 07:15:56.943145 24911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36503 every 8 connection(s)
I20260620 07:15:56.944126 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:56.946789 24912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:56.950522 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24844
I20260620 07:15:56.950677 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:56.950982 4052 external_mini_cluster.cc:1428] Setting key 464e74c13adf28fb80fe081720d229f9
I20260620 07:15:56.952787 24912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2: Bootstrap starting.
I20260620 07:15:56.955063 24912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:56.955894 24912 log.cc:824] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:56.956346 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:56.952392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33317 (local address 127.0.0.1:36503)
0620 07:15:56.952883 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:56.952901 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:56.952947 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:15:56.953661 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:56.953680 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:56.953969 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:56.954313 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:56.955130 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:56.955137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:56.955156 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:56.955170 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:56.955230 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:56.955250 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:56.955365 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:56.955467 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:15:56.955627 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":136,"threads_started":1}
I20260620 07:15:56.957801 24912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2: No bootstrap required, opened a new log
I20260620 07:15:56.959177 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:56.961994 24912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } }
I20260620 07:15:56.962282 24912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:56.962361 24912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb4fc173c4f45de884c2b2896edbba2, State: Initialized, Role: FOLLOWER
I20260620 07:15:56.962872 24912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } }
I20260620 07:15:56.963039 24912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:56.963120 24912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:56.963164 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:56.961187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36293 (local address 127.0.0.1:36503)
0620 07:15:56.961302 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:56.961305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:56.961522 (+ 217us) server_negotiation.cc:407] Connection header received
0620 07:15:56.961651 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:56.961653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:56.961681 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:56.961750 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:56.962476 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:56.962480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:56.962484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:56.962495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:56.962522 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:56.962535 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:56.962683 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:56.962884 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 07:15:56.962958 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 07:15:56.963230 24912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:56.964433 24912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } }
I20260620 07:15:56.964823 24912 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb4fc173c4f45de884c2b2896edbba2; no voters:
I20260620 07:15:56.965168 24912 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:56.965242 24922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:56.965523 24922 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [term 1 LEADER]: Becoming Leader. State: Replica: 4fb4fc173c4f45de884c2b2896edbba2, State: Running, Role: LEADER
I20260620 07:15:56.965842 24922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } }
I20260620 07:15:56.966480 24912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:56.967592 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb4fc173c4f45de884c2b2896edbba2. Latest consensus state: current_term: 1 leader_uuid: "4fb4fc173c4f45de884c2b2896edbba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } } }
I20260620 07:15:56.967720 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:56.967989 24923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb4fc173c4f45de884c2b2896edbba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb4fc173c4f45de884c2b2896edbba2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36503 } } }
I20260620 07:15:56.968075 24923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:56.968498 24931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:56.971218 24931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:56.976797 24931 catalog_manager.cc:1347] Generated new cluster ID: 15a607aa680d48859284a544556cc250
I20260620 07:15:56.976872 24931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:56.989140 24931 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:56.989506 24931 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:57.003964 24931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb4fc173c4f45de884c2b2896edbba2: Generated new TSK 0
I20260620 07:15:57.004747 24931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:57.011888 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24844
2026-06-20T07:15:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:57 dist-test-slave-ldfr krb5kdc[24943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:57 dist-test-slave-ldfr krb5kdc[24943](info): set up 2 sockets
Jun 20 07:15:57 dist-test-slave-ldfr krb5kdc[24943](info): commencing operation
krb5kdc: starting...
W20260620 07:15:58.129050 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[24943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:58.263662 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:58.375182 24956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:58.375512 24956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:58.375597 24956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:58.379704 24956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:58.379803 24956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:58.379842 24956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:58.379901 24956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:58.379941 24956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:58.385336 24956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:58.386408 24956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:58.387427 24956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:15:58.396412 24956 server_base.cc:1034] running on GCE node
W20260620 07:15:58.396593 24964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:58.396618 24962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:58.396631 24961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:58.397326 24956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:58.398448 24956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:58.399701 24956 hybrid_clock.cc:648] HybridClock initialized: now 1781939758399669 us; error 77 us; skew 500 ppm
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[24943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:58.402757 24956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:58.403717 24956 webserver.cc:460] Webserver started at http://127.0.0.1:37155/ using document root <none> and password file <none>
I20260620 07:15:58.404358 24956 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:58.404454 24956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:58.404745 24956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:58.406849 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "dc0e9d29fd4f4553ab404784903a39fd"
format_stamp: "Formatted at 2026-06-20 07:15:58 on dist-test-slave-ldfr"
server_key: "8a263d922b6176af4080e565423b809e"
server_key_iv: "7e70255a4e0660463dc59fd9460132a1"
server_key_version: "encryptionkey@0"
I20260620 07:15:58.407430 24956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "dc0e9d29fd4f4553ab404784903a39fd"
format_stamp: "Formatted at 2026-06-20 07:15:58 on dist-test-slave-ldfr"
server_key: "8a263d922b6176af4080e565423b809e"
server_key_iv: "7e70255a4e0660463dc59fd9460132a1"
server_key_version: "encryptionkey@0"
I20260620 07:15:58.412514 24956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:15:58.415256 24971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:58.416702 24956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:58.416850 24956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "dc0e9d29fd4f4553ab404784903a39fd"
format_stamp: "Formatted at 2026-06-20 07:15:58 on dist-test-slave-ldfr"
server_key: "8a263d922b6176af4080e565423b809e"
server_key_iv: "7e70255a4e0660463dc59fd9460132a1"
server_key_version: "encryptionkey@0"
I20260620 07:15:58.416985 24956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:58.434218 24956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:58.437397 24956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:58.437626 24956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:58.444766 24956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43505
I20260620 07:15:58.444796 25023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43505 every 8 connection(s)
I20260620 07:15:58.445734 24956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:58.448387 25024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:58.450517 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 24956
I20260620 07:15:58.450673 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:58.450953 4052 external_mini_cluster.cc:1428] Setting key a00c17b8014b5c856aaacf4f6811aab4
I20260620 07:15:58.455251 25024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: Bootstrap starting.
I20260620 07:15:58.457571 25024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:58.458364 25024 log.cc:824] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[24943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:15:58.460532 25024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: No bootstrap required, opened a new log
I20260620 07:15:58.464263 25024 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } }
I20260620 07:15:58.464529 25024 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:58.464587 25024 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0e9d29fd4f4553ab404784903a39fd, State: Initialized, Role: FOLLOWER
I20260620 07:15:58.465101 25024 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } }
I20260620 07:15:58.465260 25024 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:58.465359 25024 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:58.465484 25024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:58.465930 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:58.452547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:34201 (local address 127.0.0.1:43505)
0620 07:15:58.453025 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:58.453041 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:58.453084 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:15:58.453773 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:58.453791 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:58.454143 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:58.454480 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:58.455307 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:58.456297 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:58.457223 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:58.457764 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:58.457930 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:58.460836 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:58.460841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:58.460854 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:58.460880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:58.462592 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:58.463161 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:58.463165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:58.463170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:58.463290 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:58.463671 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:58.463675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:58.463677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:58.464070 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:58.464307 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:58.464603 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 07:15:58.464987 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":122,"threads_started":1}
I20260620 07:15:58.466507 25024 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } }
I20260620 07:15:58.466866 25024 leader_election.cc:302] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc0e9d29fd4f4553ab404784903a39fd; no voters:
I20260620 07:15:58.467247 25024 leader_election.cc:288] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:58.467391 25029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:58.467685 25029 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [term 1 LEADER]: Becoming Leader. State: Replica: dc0e9d29fd4f4553ab404784903a39fd, State: Running, Role: LEADER
I20260620 07:15:58.467973 25029 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } }
I20260620 07:15:58.468546 25024 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:58.469213 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:58.469214 25029 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc0e9d29fd4f4553ab404784903a39fd. Latest consensus state: current_term: 1 leader_uuid: "dc0e9d29fd4f4553ab404784903a39fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } } }
I20260620 07:15:58.469319 25029 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:58.471354 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc0e9d29fd4f4553ab404784903a39fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc0e9d29fd4f4553ab404784903a39fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43505 } } }
I20260620 07:15:58.471527 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd [sys.catalog]: This master's current role is: LEADER
W20260620 07:15:58.473940 25048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:58.474014 25048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:58.474092 25035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:58.474977 25035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:58.480372 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:58.471640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:59069 (local address 127.0.0.1:43505)
0620 07:15:58.471739 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:58.471744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:58.471759 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:15:58.471822 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:58.471825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:58.471868 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:58.471948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:58.472721 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:58.473244 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:58.474118 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:58.474403 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:58.474519 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:58.475941 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:58.475945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:58.475948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:15:58.475973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:58.477966 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:58.478548 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:58.478552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:58.478554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:58.478656 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:15:58.479069 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:15:58.479073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:15:58.479075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:15:58.479273 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:58.479784 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:58.480102 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 07:15:58.480216 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 07:15:58.481289 25035 catalog_manager.cc:1347] Generated new cluster ID: fdfe81b43fa44203ad9f25fc0ade0b7a
I20260620 07:15:58.481413 25035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:58.498688 25035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:58.499959 25035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:58.510941 25035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc0e9d29fd4f4553ab404784903a39fd: Generated new TSK 0
I20260620 07:15:58.511821 25035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:58.518132 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 24956
W20260620 07:15:58.523865 24941 connection.cc:535] client connection to 127.0.0.1:43505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43505): Connection reset by peer (error 104)
2026-06-20T07:15:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[25055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[25055](info): set up 2 sockets
Jun 20 07:15:58 dist-test-slave-ldfr krb5kdc[25055](info): commencing operation
krb5kdc: starting...
W20260620 07:15:59.573235 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:15:59 dist-test-slave-ldfr krb5kdc[25055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:15:59.699513 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:15:59.808071 25068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:15:59.808346 25068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:15:59.808391 25068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:15:59.812445 25068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:15:59.812510 25068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:15:59.812530 25068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:15:59.812546 25068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:15:59.812562 25068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:15:59.818374 25068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:15:59.819689 25068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:15:59.820955 25068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:15:59.829520 25073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:15:59.829728 25076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:59.829710 25068 server_base.cc:1034] running on GCE node
W20260620 07:15:59.829560 25074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:15:59.830555 25068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:15:59.831804 25068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:15:59.833009 25068 hybrid_clock.cc:648] HybridClock initialized: now 1781939759832993 us; error 36 us; skew 500 ppm
Jun 20 07:15:59 dist-test-slave-ldfr krb5kdc[25055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:15:59.836122 25068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:15:59.837224 25068 webserver.cc:460] Webserver started at http://127.0.0.1:38113/ using document root <none> and password file <none>
I20260620 07:15:59.837831 25068 fs_manager.cc:359] Metadata directory not provided
I20260620 07:15:59.837931 25068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:15:59.838181 25068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:15:59.840462 25068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "ad1fce85ef5142b2b1b71aa71bbce847"
format_stamp: "Formatted at 2026-06-20 07:15:59 on dist-test-slave-ldfr"
server_key: "a0baab9064b4dc7f829020c5eb44ff04"
server_key_iv: "d416655f6f4f482bbc0d66810b232bbe"
server_key_version: "encryptionkey@0"
I20260620 07:15:59.841177 25068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "ad1fce85ef5142b2b1b71aa71bbce847"
format_stamp: "Formatted at 2026-06-20 07:15:59 on dist-test-slave-ldfr"
server_key: "a0baab9064b4dc7f829020c5eb44ff04"
server_key_iv: "d416655f6f4f482bbc0d66810b232bbe"
server_key_version: "encryptionkey@0"
I20260620 07:15:59.845336 25068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:15:59.848098 25083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:15:59.849463 25068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:15:59.849643 25068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "ad1fce85ef5142b2b1b71aa71bbce847"
format_stamp: "Formatted at 2026-06-20 07:15:59 on dist-test-slave-ldfr"
server_key: "a0baab9064b4dc7f829020c5eb44ff04"
server_key_iv: "d416655f6f4f482bbc0d66810b232bbe"
server_key_version: "encryptionkey@0"
I20260620 07:15:59.849774 25068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:15:59.864184 25068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:15:59.867630 25068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:15:59.867867 25068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:15:59.875365 25135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34647 every 8 connection(s)
I20260620 07:15:59.875365 25068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34647
I20260620 07:15:59.876420 25068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:15:59.879420 25136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:15:59.885450 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: Bootstrap starting.
I20260620 07:15:59.886020 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25068
I20260620 07:15:59.886160 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:15:59.886442 4052 external_mini_cluster.cc:1428] Setting key 8a9081ba4e9ef655a8ba0aefc16ed52e
I20260620 07:15:59.888334 25136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: Neither blocks nor log segments found. Creating new log.
I20260620 07:15:59.889266 25136 log.cc:824] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: Log is configured to *not* fsync() on all Append() calls
I20260620 07:15:59.891525 25136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: No bootstrap required, opened a new log
I20260620 07:15:59.895939 25136 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260620 07:15:59.896236 25136 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:15:59.896286 25136 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1fce85ef5142b2b1b71aa71bbce847, State: Initialized, Role: FOLLOWER
I20260620 07:15:59.896813 25136 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260620 07:15:59.896981 25136 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:15:59.897042 25136 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:15:59.897140 25136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:15:59.897140 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:59.887907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:60437 (local address 127.0.0.1:34647)
0620 07:15:59.888381 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:59.888403 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:59.888445 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:15:59.889400 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:59.889429 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:59.889856 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:59.890215 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:59.890637 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:59.892091 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:59.893070 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:59.893794 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:59.894020 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:59.894922 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:59.894929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:59.894947 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:59.895507 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:59.895578 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:59.895608 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:59.895738 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:59.895947 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:15:59.896091 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":123,"threads_started":1}
I20260620 07:15:59.898382 25136 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260620 07:15:59.898772 25136 leader_election.cc:302] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad1fce85ef5142b2b1b71aa71bbce847; no voters:
I20260620 07:15:59.899101 25136 leader_election.cc:288] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:15:59.899251 25141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:15:59.899501 25141 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [term 1 LEADER]: Becoming Leader. State: Replica: ad1fce85ef5142b2b1b71aa71bbce847, State: Running, Role: LEADER
I20260620 07:15:59.899840 25141 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } }
I20260620 07:15:59.900517 25136 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:15:59.900894 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:15:59.901733 25142 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } } }
I20260620 07:15:59.901865 25142 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [sys.catalog]: This master's current role is: LEADER
I20260620 07:15:59.902298 25143 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1fce85ef5142b2b1b71aa71bbce847. Latest consensus state: current_term: 1 leader_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1fce85ef5142b2b1b71aa71bbce847" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34647 } } }
I20260620 07:15:59.902449 25143 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847 [sys.catalog]: This master's current role is: LEADER
W20260620 07:15:59.905964 25160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:15:59.906057 25160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:15:59.906126 25150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:15:59.906857 25150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:15:59.909861 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:15:59.905343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:35213 (local address 127.0.0.1:34647)
0620 07:15:59.905486 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 07:15:59.905491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:15:59.905888 (+ 397us) server_negotiation.cc:407] Connection header received
0620 07:15:59.906019 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:15:59.906022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:15:59.906082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:15:59.906178 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:15:59.906573 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:59.907185 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:59.908090 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:15:59.908328 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:15:59.908438 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:15:59.909397 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:15:59.909403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:15:59.909411 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:15:59.909424 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:15:59.909450 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:15:59.909460 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:15:59.909559 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:15:59.909680 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 07:15:59.909715 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:15:59.912695 25150 catalog_manager.cc:1347] Generated new cluster ID: 1ff665bc68914140aa5ee07277274765
I20260620 07:15:59.912767 25150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:15:59.935752 25150 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:15:59.937003 25150 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:15:59.944269 25150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad1fce85ef5142b2b1b71aa71bbce847: Generated new TSK 0
I20260620 07:15:59.945199 25150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:15:59.949736 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25068
W20260620 07:15:59.954747 25053 connection.cc:535] client connection to 127.0.0.1:34647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34647): Connection reset by peer (error 104)
2026-06-20T07:15:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:00 dist-test-slave-ldfr krb5kdc[25167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:00 dist-test-slave-ldfr krb5kdc[25167](info): set up 2 sockets
Jun 20 07:16:00 dist-test-slave-ldfr krb5kdc[25167](info): commencing operation
krb5kdc: starting...
W20260620 07:16:01.001097 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:01 dist-test-slave-ldfr krb5kdc[25167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:16:01.137449 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:01.255923 25180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:01.256309 25180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:01.256399 25180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:01.260684 25180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:01.260800 25180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:01.260838 25180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:01.260878 25180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:01.260912 25180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:01.266598 25180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:01.267817 25180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:01.269021 25180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:01.278209 25188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:01.278209 25186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:01.278227 25185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:01.278474 25180 server_base.cc:1034] running on GCE node
I20260620 07:16:01.279058 25180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:01.280296 25180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:01.281603 25180 hybrid_clock.cc:648] HybridClock initialized: now 1781939761281576 us; error 45 us; skew 500 ppm
Jun 20 07:16:01 dist-test-slave-ldfr krb5kdc[25167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:01.284955 25180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:16:01.285962 25180 webserver.cc:460] Webserver started at http://127.0.0.1:42491/ using document root <none> and password file <none>
I20260620 07:16:01.286624 25180 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:01.286722 25180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:01.286986 25180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:01.289261 25180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "d8c3f33588114cc591b89ef11f7423ae"
format_stamp: "Formatted at 2026-06-20 07:16:01 on dist-test-slave-ldfr"
server_key: "dbe574eae33fca182433281bb8904289"
server_key_iv: "4161d781cef255511ce5da196cf51384"
server_key_version: "encryptionkey@0"
I20260620 07:16:01.289892 25180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "d8c3f33588114cc591b89ef11f7423ae"
format_stamp: "Formatted at 2026-06-20 07:16:01 on dist-test-slave-ldfr"
server_key: "dbe574eae33fca182433281bb8904289"
server_key_iv: "4161d781cef255511ce5da196cf51384"
server_key_version: "encryptionkey@0"
I20260620 07:16:01.294157 25180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:16:01.297374 25195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:01.298883 25180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:16:01.299075 25180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "d8c3f33588114cc591b89ef11f7423ae"
format_stamp: "Formatted at 2026-06-20 07:16:01 on dist-test-slave-ldfr"
server_key: "dbe574eae33fca182433281bb8904289"
server_key_iv: "4161d781cef255511ce5da196cf51384"
server_key_version: "encryptionkey@0"
I20260620 07:16:01.299230 25180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:01.304241 25180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:01.307341 25180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:01.307564 25180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:01.315491 25180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35059
I20260620 07:16:01.315516 25247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35059 every 8 connection(s)
I20260620 07:16:01.316509 25180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:01.319525 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:01.324404 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25180
I20260620 07:16:01.324568 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:01.324890 4052 external_mini_cluster.cc:1428] Setting key f1cf5ec0c915e0320e19023192ba68a3
I20260620 07:16:01.326239 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae: Bootstrap starting.
I20260620 07:16:01.328950 25248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:01.329780 25248 log.cc:824] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:01.330780 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:01.326563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:41349 (local address 127.0.0.1:35059)
0620 07:16:01.327049 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:01.327065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:01.327106 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:16:01.327818 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:01.327836 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:01.328258 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:01.328735 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:01.329478 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:01.329482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:01.329494 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:01.329504 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:01.329552 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:01.329571 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:01.329700 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:01.329814 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 07:16:01.329991 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":150,"threads_started":1}
I20260620 07:16:01.332029 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae: No bootstrap required, opened a new log
I20260620 07:16:01.333931 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:01.336549 25248 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } }
I20260620 07:16:01.336901 25248 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:01.336971 25248 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c3f33588114cc591b89ef11f7423ae, State: Initialized, Role: FOLLOWER
I20260620 07:16:01.337471 25248 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } }
I20260620 07:16:01.337630 25248 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:01.337720 25248 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:01.337833 25248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:01.338414 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:01.336345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:56941 (local address 127.0.0.1:35059)
0620 07:16:01.336466 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:01.336471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:01.336573 (+ 102us) server_negotiation.cc:407] Connection header received
0620 07:16:01.336809 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:01.336813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:01.336859 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:01.336946 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:01.337786 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:01.337789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:01.337792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:01.337805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:01.337848 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:01.337865 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:01.337966 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:01.338158 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:16:01.338263 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:16:01.339128 25248 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } }
I20260620 07:16:01.339540 25248 leader_election.cc:302] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c3f33588114cc591b89ef11f7423ae; no voters:
I20260620 07:16:01.339905 25248 leader_election.cc:288] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:01.340025 25258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:01.340307 25258 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [term 1 LEADER]: Becoming Leader. State: Replica: d8c3f33588114cc591b89ef11f7423ae, State: Running, Role: LEADER
I20260620 07:16:01.340660 25258 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } }
I20260620 07:16:01.341423 25248 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:01.342582 25260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c3f33588114cc591b89ef11f7423ae. Latest consensus state: current_term: 1 leader_uuid: "d8c3f33588114cc591b89ef11f7423ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } } }
I20260620 07:16:01.342736 25260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:01.342592 25259 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c3f33588114cc591b89ef11f7423ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c3f33588114cc591b89ef11f7423ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35059 } } }
I20260620 07:16:01.342989 25259 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:01.343231 25267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:01.346091 25267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:01.351671 25267 catalog_manager.cc:1347] Generated new cluster ID: 0b8c7b8324ba40d7a22ce103b501f9e5
I20260620 07:16:01.351763 25267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:01.364372 25267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:01.364787 25267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:01.377718 25267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c3f33588114cc591b89ef11f7423ae: Generated new TSK 0
I20260620 07:16:01.378512 25267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:01.384351 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25180
2026-06-20T07:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:01 dist-test-slave-ldfr krb5kdc[25279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:01 dist-test-slave-ldfr krb5kdc[25279](info): set up 2 sockets
Jun 20 07:16:01 dist-test-slave-ldfr krb5kdc[25279](info): commencing operation
krb5kdc: starting...
W20260620 07:16:02.457120 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:16:02.596856 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:02.713197 25292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:02.713478 25292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:02.713521 25292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:02.717566 25292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:02.717638 25292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:02.717656 25292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:02.717674 25292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:02.717695 25292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:02.723023 25292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:02.724108 25292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:02.725205 25292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:02.734061 25298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:02.734324 25292 server_base.cc:1034] running on GCE node
W20260620 07:16:02.734061 25297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:02.734190 25300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:02.735055 25292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:02.736243 25292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:02.737460 25292 hybrid_clock.cc:648] HybridClock initialized: now 1781939762737410 us; error 63 us; skew 500 ppm
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:02.740489 25292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:16:02.741554 25292 webserver.cc:460] Webserver started at http://127.0.0.1:34013/ using document root <none> and password file <none>
I20260620 07:16:02.742157 25292 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:02.742260 25292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:02.742517 25292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:02.745014 25292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "1629561cb9074bd48aed498fdbcaea53"
format_stamp: "Formatted at 2026-06-20 07:16:02 on dist-test-slave-ldfr"
server_key: "ca90b810b2b9af5f81b87c8d0ee31a6e"
server_key_iv: "cff3f539bd4e8181b32f333a2b5bb5c0"
server_key_version: "encryptionkey@0"
I20260620 07:16:02.745637 25292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "1629561cb9074bd48aed498fdbcaea53"
format_stamp: "Formatted at 2026-06-20 07:16:02 on dist-test-slave-ldfr"
server_key: "ca90b810b2b9af5f81b87c8d0ee31a6e"
server_key_iv: "cff3f539bd4e8181b32f333a2b5bb5c0"
server_key_version: "encryptionkey@0"
I20260620 07:16:02.749926 25292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:16:02.752780 25307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:02.754113 25292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:16:02.754272 25292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "1629561cb9074bd48aed498fdbcaea53"
format_stamp: "Formatted at 2026-06-20 07:16:02 on dist-test-slave-ldfr"
server_key: "ca90b810b2b9af5f81b87c8d0ee31a6e"
server_key_iv: "cff3f539bd4e8181b32f333a2b5bb5c0"
server_key_version: "encryptionkey@0"
I20260620 07:16:02.754403 25292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:02.770629 25292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:02.774101 25292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:02.774350 25292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:02.782008 25292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46169
I20260620 07:16:02.782025 25359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46169 every 8 connection(s)
I20260620 07:16:02.783030 25292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:02.783877 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25292
I20260620 07:16:02.784019 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:02.784325 4052 external_mini_cluster.cc:1428] Setting key e0ba923a98938575ab9256a724c93044
I20260620 07:16:02.786752 25360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:02.794034 25360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53: Bootstrap starting.
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:16:02.796680 25360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:02.797534 25360 log.cc:824] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:02.800001 25360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53: No bootstrap required, opened a new log
I20260620 07:16:02.800725 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:02.785969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:57711 (local address 127.0.0.1:46169)
0620 07:16:02.786461 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:02.786476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:02.786512 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:16:02.787263 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:02.787282 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:02.787645 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:02.787922 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:02.789053 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:02.790102 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:02.791354 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:02.791824 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:02.791955 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:02.795280 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:02.795285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:02.795297 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:02.795324 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:02.797331 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:02.798023 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:02.798033 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:02.798040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:02.798191 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:02.798570 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:02.798572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:02.798574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:02.799012 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:02.799184 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:02.799534 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:16:02.799833 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":164,"threads_started":1}
I20260620 07:16:02.804282 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:02.804872 25360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260620 07:16:02.805218 25360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:02.805300 25360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1629561cb9074bd48aed498fdbcaea53, State: Initialized, Role: FOLLOWER
I20260620 07:16:02.805856 25360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260620 07:16:02.806056 25360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:02.806139 25360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:02.806263 25360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:02.807540 25360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260620 07:16:02.807932 25360 leader_election.cc:302] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1629561cb9074bd48aed498fdbcaea53; no voters:
I20260620 07:16:02.808298 25360 leader_election.cc:288] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:02.808373 25370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:02.808599 25370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [term 1 LEADER]: Becoming Leader. State: Replica: 1629561cb9074bd48aed498fdbcaea53, State: Running, Role: LEADER
I20260620 07:16:02.809077 25370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260620 07:16:02.809880 25360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:02.810437 25370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1629561cb9074bd48aed498fdbcaea53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260620 07:16:02.810623 25370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:02.811141 25377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:02.810850 25371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1629561cb9074bd48aed498fdbcaea53. Latest consensus state: current_term: 1 leader_uuid: "1629561cb9074bd48aed498fdbcaea53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1629561cb9074bd48aed498fdbcaea53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260620 07:16:02.811419 25371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:02.815145 25377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:02.816344 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:02.806826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:35321 (local address 127.0.0.1:46169)
0620 07:16:02.806994 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:02.806998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:02.807115 (+ 117us) server_negotiation.cc:407] Connection header received
0620 07:16:02.807251 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:02.807254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:02.807304 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:02.807396 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:02.808291 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:02.809049 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:02.809904 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:02.810196 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:02.810311 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:02.812354 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:02.812358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:02.812363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:02.812392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:02.814043 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:02.814607 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:02.814611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:02.814613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:02.814714 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:02.815082 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:02.815086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:02.815088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:02.815344 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:02.815506 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:02.816099 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 07:16:02.816197 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 07:16:02.822059 25377 catalog_manager.cc:1347] Generated new cluster ID: 2009318aaf36411db8b1a03b842cdacd
I20260620 07:16:02.822161 25377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:02.827621 25377 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:02.828884 25377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:02.847975 25377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1629561cb9074bd48aed498fdbcaea53: Generated new TSK 0
I20260620 07:16:02.848932 25377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:02.859838 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25292
W20260620 07:16:02.865533 25277 connection.cc:535] client connection to 127.0.0.1:46169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46169): Connection reset by peer (error 104)
2026-06-20T07:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25391](info): set up 2 sockets
Jun 20 07:16:02 dist-test-slave-ldfr krb5kdc[25391](info): commencing operation
krb5kdc: starting...
W20260620 07:16:03.953346 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:04 dist-test-slave-ldfr krb5kdc[25391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:16:04.088022 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:04.200266 25404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:04.200562 25404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:04.200605 25404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:04.205008 25404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:04.205087 25404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:04.205108 25404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:04.205127 25404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:04.205145 25404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:04.210734 25404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39515
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:04.211812 25404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:04.212811 25404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:04.221583 25409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:04.221583 25412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:04.221730 25410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:04.221920 25404 server_base.cc:1034] running on GCE node
I20260620 07:16:04.222549 25404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:04.223790 25404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:04.224987 25404 hybrid_clock.cc:648] HybridClock initialized: now 1781939764224943 us; error 56 us; skew 500 ppm
Jun 20 07:16:04 dist-test-slave-ldfr krb5kdc[25391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:04.232139 25404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:16:04.233214 25404 webserver.cc:460] Webserver started at http://127.0.0.1:34543/ using document root <none> and password file <none>
I20260620 07:16:04.233776 25404 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:04.233839 25404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:04.234010 25404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:04.236181 25404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "8248ea0c16dd4c289622f356a37b602d"
format_stamp: "Formatted at 2026-06-20 07:16:04 on dist-test-slave-ldfr"
server_key: "2772e1018f05b04c59ef59b82bd53130"
server_key_iv: "4168e699ff7d1a789d7a94400b602cb6"
server_key_version: "encryptionkey@0"
I20260620 07:16:04.236799 25404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "8248ea0c16dd4c289622f356a37b602d"
format_stamp: "Formatted at 2026-06-20 07:16:04 on dist-test-slave-ldfr"
server_key: "2772e1018f05b04c59ef59b82bd53130"
server_key_iv: "4168e699ff7d1a789d7a94400b602cb6"
server_key_version: "encryptionkey@0"
I20260620 07:16:04.240756 25404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:16:04.243387 25419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:04.244995 25404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:04.245152 25404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "8248ea0c16dd4c289622f356a37b602d"
format_stamp: "Formatted at 2026-06-20 07:16:04 on dist-test-slave-ldfr"
server_key: "2772e1018f05b04c59ef59b82bd53130"
server_key_iv: "4168e699ff7d1a789d7a94400b602cb6"
server_key_version: "encryptionkey@0"
I20260620 07:16:04.245281 25404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:04.266281 25404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:04.269783 25404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:04.270020 25404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:04.278136 25404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39515
I20260620 07:16:04.278156 25471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39515 every 8 connection(s)
I20260620 07:16:04.279204 25404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:04.282291 25472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:04.285739 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25404
I20260620 07:16:04.285879 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:04.286173 4052 external_mini_cluster.cc:1428] Setting key 0d58cb2ba52f9a6673c5739201ff1b1a
I20260620 07:16:04.288663 25472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d: Bootstrap starting.
I20260620 07:16:04.291050 25472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:04.291844 25472 log.cc:824] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:04.294065 25472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d: No bootstrap required, opened a new log
I20260620 07:16:04.295758 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:04.287675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:50943 (local address 127.0.0.1:39515)
0620 07:16:04.288131 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:04.288147 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:04.288180 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:16:04.288952 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:04.288971 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:04.289336 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:04.289666 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:04.290139 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:04.291163 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:04.292140 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:04.292858 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:04.293027 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:04.293998 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:04.294003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:04.294017 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:04.294478 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:04.294532 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:04.294555 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:04.294698 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:04.294883 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 07:16:04.295031 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260620 07:16:04.298332 25472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260620 07:16:04.298640 25472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:04.298699 25472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8248ea0c16dd4c289622f356a37b602d, State: Initialized, Role: FOLLOWER
I20260620 07:16:04.299052 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:04.299229 25472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260620 07:16:04.299392 25472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:04.299461 25472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:04.299567 25472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:04.300645 25472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260620 07:16:04.301008 25472 leader_election.cc:302] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8248ea0c16dd4c289622f356a37b602d; no voters:
I20260620 07:16:04.301355 25472 leader_election.cc:288] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:04.301429 25481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:04.301651 25481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [term 1 LEADER]: Becoming Leader. State: Replica: 8248ea0c16dd4c289622f356a37b602d, State: Running, Role: LEADER
I20260620 07:16:04.302012 25481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } }
I20260620 07:16:04.302712 25472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:04.303644 25483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8248ea0c16dd4c289622f356a37b602d. Latest consensus state: current_term: 1 leader_uuid: "8248ea0c16dd4c289622f356a37b602d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260620 07:16:04.303853 25483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:04.304123 25483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8248ea0c16dd4c289622f356a37b602d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8248ea0c16dd4c289622f356a37b602d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39515 } } }
I20260620 07:16:04.304260 25483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:04.305042 25491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:04.306149 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:04.301291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:55667 (local address 127.0.0.1:39515)
0620 07:16:04.301412 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:04.301416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:04.301828 (+ 412us) server_negotiation.cc:407] Connection header received
0620 07:16:04.301864 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:04.301867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:04.301914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:04.301981 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:04.302379 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:04.302996 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:04.304118 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:04.304440 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:04.304538 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:04.305600 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:04.305604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:04.305607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:04.305622 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:04.305696 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:04.305714 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:04.305829 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:04.305958 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 07:16:04.305993 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:04.308300 25491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:04.314488 25491 catalog_manager.cc:1347] Generated new cluster ID: 08f7042cf8964c5eba6a851c31bdd6ca
I20260620 07:16:04.314636 25491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:04.330332 25491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:04.331650 25491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:04.340732 25491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8248ea0c16dd4c289622f356a37b602d: Generated new TSK 0
I20260620 07:16:04.341614 25491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:04.345882 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25404
W20260620 07:16:04.351006 25389 connection.cc:535] client connection to 127.0.0.1:39515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39515): Connection reset by peer (error 104)
2026-06-20T07:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:04 dist-test-slave-ldfr krb5kdc[25503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:04 dist-test-slave-ldfr krb5kdc[25503](info): set up 2 sockets
Jun 20 07:16:04 dist-test-slave-ldfr krb5kdc[25503](info): commencing operation
krb5kdc: starting...
W20260620 07:16:05.388810 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:05 dist-test-slave-ldfr krb5kdc[25503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:16:05.513792 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:05.628468 25516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:05.628757 25516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:05.628801 25516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:05.632896 25516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:05.632967 25516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:05.632987 25516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:05.633005 25516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:05.633020 25516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:05.638456 25516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:05.639709 25516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:05.640712 25516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:05.649444 25521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:05.649431 25522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:05.649482 25516 server_base.cc:1034] running on GCE node
W20260620 07:16:05.649410 25524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:05.650213 25516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:05.651535 25516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:05.652735 25516 hybrid_clock.cc:648] HybridClock initialized: now 1781939765652723 us; error 56 us; skew 500 ppm
Jun 20 07:16:05 dist-test-slave-ldfr krb5kdc[25503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:05.655532 25516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:16:05.656445 25516 webserver.cc:460] Webserver started at http://127.0.0.1:41383/ using document root <none> and password file <none>
I20260620 07:16:05.657073 25516 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:05.657166 25516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:05.657392 25516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:05.659415 25516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "f300af615cbe4d0883e16c29df8cf5bf"
format_stamp: "Formatted at 2026-06-20 07:16:05 on dist-test-slave-ldfr"
server_key: "88be69d6ce82c404fb1ef54cf2b2b444"
server_key_iv: "d9c955d4b805341f83213a0c2ef881a2"
server_key_version: "encryptionkey@0"
I20260620 07:16:05.659961 25516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "f300af615cbe4d0883e16c29df8cf5bf"
format_stamp: "Formatted at 2026-06-20 07:16:05 on dist-test-slave-ldfr"
server_key: "88be69d6ce82c404fb1ef54cf2b2b444"
server_key_iv: "d9c955d4b805341f83213a0c2ef881a2"
server_key_version: "encryptionkey@0"
I20260620 07:16:05.663587 25516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:05.665930 25531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:05.667121 25516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:05.667289 25516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "f300af615cbe4d0883e16c29df8cf5bf"
format_stamp: "Formatted at 2026-06-20 07:16:05 on dist-test-slave-ldfr"
server_key: "88be69d6ce82c404fb1ef54cf2b2b444"
server_key_iv: "d9c955d4b805341f83213a0c2ef881a2"
server_key_version: "encryptionkey@0"
I20260620 07:16:05.667404 25516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:05.671922 25516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:05.675222 25516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:05.675419 25516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:05.683283 25516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39199
I20260620 07:16:05.683290 25583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39199 every 8 connection(s)
I20260620 07:16:05.684300 25516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:05.687155 25584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:05.690179 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25516
I20260620 07:16:05.690361 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:05.690653 4052 external_mini_cluster.cc:1428] Setting key a29443fce4a8ee2ed134df66d8989e6e
I20260620 07:16:05.693125 25584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: Bootstrap starting.
I20260620 07:16:05.695494 25584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:05.696255 25584 log.cc:824] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:05.696432 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:05.692185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:36823 (local address 127.0.0.1:39199)
0620 07:16:05.692736 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:05.692752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:05.692790 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:16:05.693535 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:05.693555 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:05.693860 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:05.694226 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:05.695058 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:05.695063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:05.695076 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:05.695086 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:05.695140 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:05.695160 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:05.695286 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:05.695395 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 07:16:05.695614 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260620 07:16:05.698390 25584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: No bootstrap required, opened a new log
I20260620 07:16:05.699725 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:05.702554 25584 raft_consensus.cc:348] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 07:16:05.702847 25584 raft_consensus.cc:374] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:05.702921 25584 raft_consensus.cc:729] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f300af615cbe4d0883e16c29df8cf5bf, State: Initialized, Role: FOLLOWER
I20260620 07:16:05.703473 25584 consensus_queue.cc:260] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 07:16:05.703629 25584 raft_consensus.cc:388] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:05.703713 25584 raft_consensus.cc:482] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:05.703831 25584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:05.704246 25587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:05.702043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:58265 (local address 127.0.0.1:39199)
0620 07:16:05.702185 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:05.702190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:05.702349 (+ 159us) server_negotiation.cc:407] Connection header received
0620 07:16:05.702461 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:05.702463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:05.702500 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:05.702591 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:05.703756 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:05.703760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:05.703762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:05.703784 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:05.703835 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:05.703851 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:05.703950 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:05.704068 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:16:05.704114 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:16:05.705039 25584 raft_consensus.cc:504] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 07:16:05.705408 25584 leader_election.cc:302] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f300af615cbe4d0883e16c29df8cf5bf; no voters:
I20260620 07:16:05.705739 25584 leader_election.cc:288] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:05.705869 25594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:05.706130 25594 raft_consensus.cc:686] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [term 1 LEADER]: Becoming Leader. State: Replica: f300af615cbe4d0883e16c29df8cf5bf, State: Running, Role: LEADER
I20260620 07:16:05.706475 25594 consensus_queue.cc:237] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } }
I20260620 07:16:05.707022 25584 sys_catalog.cc:564] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:05.708190 25595 sys_catalog.cc:455] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f300af615cbe4d0883e16c29df8cf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260620 07:16:05.708175 25596 sys_catalog.cc:455] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f300af615cbe4d0883e16c29df8cf5bf. Latest consensus state: current_term: 1 leader_uuid: "f300af615cbe4d0883e16c29df8cf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f300af615cbe4d0883e16c29df8cf5bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39199 } } }
I20260620 07:16:05.708369 25596 sys_catalog.cc:458] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:05.708354 25595 sys_catalog.cc:458] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf [sys.catalog]: This master's current role is: LEADER
W20260620 07:16:05.711911 25609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:05.712002 25609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:05.712077 25603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:05.712972 25603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:05.718587 25603 catalog_manager.cc:1347] Generated new cluster ID: 0cac51cdb0fa49e6851bcedb483a39e0
I20260620 07:16:05.718663 25603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:05.757654 25603 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:05.758025 25603 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:05.763295 25603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f300af615cbe4d0883e16c29df8cf5bf: Generated new TSK 0
I20260620 07:16:05.764065 25603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:05.777585 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25516
2026-06-20T07:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:05 dist-test-slave-ldfr krb5kdc[25615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:05 dist-test-slave-ldfr krb5kdc[25615](info): set up 2 sockets
Jun 20 07:16:05 dist-test-slave-ldfr krb5kdc[25615](info): commencing operation
krb5kdc: starting...
W20260620 07:16:06.817056 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:06 dist-test-slave-ldfr krb5kdc[25615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:06.943740 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:42959
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:07.055210 25628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:07.055644 25628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:07.055737 25628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:07.059902 25628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:07.060012 25628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:07.060052 25628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:07.060091 25628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:07.060122 25628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:07.065773 25628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:42959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:07.066956 25628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:07.068033 25628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:07.076890 25633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:07.076894 25636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:07.076993 25628 server_base.cc:1034] running on GCE node
W20260620 07:16:07.076890 25634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:07.077777 25628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:07.078934 25628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:07.080153 25628 hybrid_clock.cc:648] HybridClock initialized: now 1781939767080127 us; error 43 us; skew 500 ppm
Jun 20 07:16:07 dist-test-slave-ldfr krb5kdc[25615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:07.083210 25628 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:07.084192 25628 webserver.cc:460] Webserver started at http://127.3.245.62:41031/ using document root <none> and password file <none>
I20260620 07:16:07.084833 25628 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:07.084903 25628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:07.085127 25628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:07.087340 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "6f461f13e7764a71b2751a1b918d002f"
format_stamp: "Formatted at 2026-06-20 07:16:07 on dist-test-slave-ldfr"
server_key: "0c6a3271551692d84ffdcf2b4c710609"
server_key_iv: "6013e77015cd1e5951f86408d9fc6e64"
server_key_version: "encryptionkey@0"
I20260620 07:16:07.087949 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "6f461f13e7764a71b2751a1b918d002f"
format_stamp: "Formatted at 2026-06-20 07:16:07 on dist-test-slave-ldfr"
server_key: "0c6a3271551692d84ffdcf2b4c710609"
server_key_iv: "6013e77015cd1e5951f86408d9fc6e64"
server_key_version: "encryptionkey@0"
I20260620 07:16:07.092046 25628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:16:07.094800 25643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:07.096299 25628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:16:07.096491 25628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "6f461f13e7764a71b2751a1b918d002f"
format_stamp: "Formatted at 2026-06-20 07:16:07 on dist-test-slave-ldfr"
server_key: "0c6a3271551692d84ffdcf2b4c710609"
server_key_iv: "6013e77015cd1e5951f86408d9fc6e64"
server_key_version: "encryptionkey@0"
I20260620 07:16:07.096671 25628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:07.111773 25628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:07.115540 25628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:07.115799 25628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:07.123514 25628 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:42959
I20260620 07:16:07.123593 25695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:42959 every 8 connection(s)
I20260620 07:16:07.124701 25628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:07.127818 25696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:07.130419 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25628
I20260620 07:16:07.130596 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:07.130895 4052 external_mini_cluster.cc:1428] Setting key 2640185b7f3cb8f265d7e501665b2c23
I20260620 07:16:07.134231 25696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f: Bootstrap starting.
I20260620 07:16:07.136579 25696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:07.137413 25696 log.cc:824] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:07.139942 25696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f: No bootstrap required, opened a new log
Jun 20 07:16:07 dist-test-slave-ldfr krb5kdc[25615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:07.144963 25696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } }
I20260620 07:16:07.145270 25696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:07.145328 25696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f461f13e7764a71b2751a1b918d002f, State: Initialized, Role: FOLLOWER
I20260620 07:16:07.145905 25696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } }
I20260620 07:16:07.146106 25696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:07.146183 25696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:07.146277 25696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:07.147323 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:07.132423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56733 (local address 127.3.245.62:42959)
0620 07:16:07.132960 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:07.132976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:07.133021 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:16:07.133695 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:07.133713 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:07.134062 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:07.134411 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:07.135273 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:07.136326 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:07.137314 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:07.137928 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:07.138096 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:07.141135 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:07.141142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:07.141160 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:07.141200 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:07.143477 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:07.144056 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:07.144065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:07.144072 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:07.144223 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:07.144607 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:07.144628 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:07.144630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:07.145183 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:07.145430 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:07.145872 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 07:16:07.146123 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":192,"threads_started":1}
I20260620 07:16:07.147694 25696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } }
I20260620 07:16:07.148051 25696 leader_election.cc:302] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f461f13e7764a71b2751a1b918d002f; no voters:
I20260620 07:16:07.148387 25696 leader_election.cc:288] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:07.148518 25701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:07.148793 25701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [term 1 LEADER]: Becoming Leader. State: Replica: 6f461f13e7764a71b2751a1b918d002f, State: Running, Role: LEADER
I20260620 07:16:07.149155 25701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } }
I20260620 07:16:07.149780 25696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:07.150424 25701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f461f13e7764a71b2751a1b918d002f. Latest consensus state: current_term: 1 leader_uuid: "6f461f13e7764a71b2751a1b918d002f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } } }
I20260620 07:16:07.150578 25701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:07.150964 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:07.150990 25703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f461f13e7764a71b2751a1b918d002f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f461f13e7764a71b2751a1b918d002f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 42959 } } }
I20260620 07:16:07.151090 25703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:07.151253 25707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:07.154402 25707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:07.160746 25707 catalog_manager.cc:1347] Generated new cluster ID: 841aa59210c2465fb14b3e3db5788ea9
I20260620 07:16:07.160871 25707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:07.163828 25699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:07.153895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47811 (local address 127.3.245.62:42959)
0620 07:16:07.154018 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:07.154023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:07.154564 (+ 541us) server_negotiation.cc:407] Connection header received
0620 07:16:07.154722 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:07.154725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:07.154775 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:07.154861 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:07.155684 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:07.156447 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:07.157303 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:07.157572 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:07.157685 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:07.159341 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:07.159344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:07.159348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:07.159374 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:07.161758 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:07.162369 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:07.162373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:07.162375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:07.162468 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:07.162811 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:07.162814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:07.162816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:07.163041 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:07.163209 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:07.163449 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 07:16:07.163608 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:16:07.192384 25707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:07.193791 25707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:07.199009 25707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f461f13e7764a71b2751a1b918d002f: Generated new TSK 0
I20260620 07:16:07.199879 25707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:07.206375 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25628
W20260620 07:16:07.212093 25613 connection.cc:535] client connection to 127.3.245.62:42959 recv error: Network error: failed to read from TLS socket (remote: 127.3.245.62:42959): Connection reset by peer (error 104)
2026-06-20T07:16:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:07 dist-test-slave-ldfr krb5kdc[25727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:07 dist-test-slave-ldfr krb5kdc[25727](info): set up 2 sockets
Jun 20 07:16:07 dist-test-slave-ldfr krb5kdc[25727](info): commencing operation
krb5kdc: starting...
W20260620 07:16:08.241093 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:08 dist-test-slave-ldfr krb5kdc[25727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:08.363536 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41457
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:37813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:41457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:08.477594 25740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:08.477921 25740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:08.478003 25740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:08.482180 25740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:08.482278 25740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:08.482316 25740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:08.482354 25740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:08.482393 25740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:08.487865 25740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:37813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:41457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:08.489056 25740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:08.490072 25740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:08.498854 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:08.498870 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:08.498934 25740 server_base.cc:1034] running on GCE node
W20260620 07:16:08.498871 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:08.499786 25740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:08.501072 25740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:08.502270 25740 hybrid_clock.cc:648] HybridClock initialized: now 1781939768502256 us; error 26 us; skew 500 ppm
Jun 20 07:16:08 dist-test-slave-ldfr krb5kdc[25727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:08.505227 25740 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:08.506176 25740 webserver.cc:460] Webserver started at http://127.3.245.62:37685/ using document root <none> and password file <none>
I20260620 07:16:08.506755 25740 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:08.506846 25740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:08.507091 25740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:08.509282 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "18bf25cde111436c9df31ba6e63a5c7e"
format_stamp: "Formatted at 2026-06-20 07:16:08 on dist-test-slave-ldfr"
server_key: "cabce28d89c781062763e44b08f4f19f"
server_key_iv: "32bd782310d414a1ae62b47bfe1dc454"
server_key_version: "encryptionkey@0"
I20260620 07:16:08.509868 25740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "18bf25cde111436c9df31ba6e63a5c7e"
format_stamp: "Formatted at 2026-06-20 07:16:08 on dist-test-slave-ldfr"
server_key: "cabce28d89c781062763e44b08f4f19f"
server_key_iv: "32bd782310d414a1ae62b47bfe1dc454"
server_key_version: "encryptionkey@0"
I20260620 07:16:08.513851 25740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:16:08.516484 25755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:08.517923 25740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:08.518090 25740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "18bf25cde111436c9df31ba6e63a5c7e"
format_stamp: "Formatted at 2026-06-20 07:16:08 on dist-test-slave-ldfr"
server_key: "cabce28d89c781062763e44b08f4f19f"
server_key_iv: "32bd782310d414a1ae62b47bfe1dc454"
server_key_version: "encryptionkey@0"
I20260620 07:16:08.518215 25740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:08.529560 25740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:08.532294 25740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:08.532500 25740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:08.539511 25740 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:41457
I20260620 07:16:08.539621 25807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:41457 every 8 connection(s)
I20260620 07:16:08.540511 25740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:08.543339 25808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:08.548816 25808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: Bootstrap starting.
I20260620 07:16:08.549937 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25740
I20260620 07:16:08.550089 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:08.550350 4052 external_mini_cluster.cc:1428] Setting key e096c8a7a3edab2c0d49ce6122dedbb5
I20260620 07:16:08.551600 25808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:08.552326 25808 log.cc:824] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:08.554425 25808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: No bootstrap required, opened a new log
I20260620 07:16:08.558770 25808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } }
I20260620 07:16:08.559034 25808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:08.559115 25808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18bf25cde111436c9df31ba6e63a5c7e, State: Initialized, Role: FOLLOWER
I20260620 07:16:08.559535 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:08.551492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38139 (local address 127.3.245.62:41457)
0620 07:16:08.551961 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:08.551977 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:08.552012 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:16:08.552793 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:08.552813 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:08.553153 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:08.553436 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:08.553765 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:08.554768 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:08.555963 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:08.556433 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:08.556552 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:08.557521 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:08.557526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:08.557539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:08.557967 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:08.558017 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:08.558039 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:08.558142 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:08.558360 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:16:08.558531 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":164,"threads_started":1}
I20260620 07:16:08.559559 25808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } }
I20260620 07:16:08.559755 25808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:08.559828 25808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:08.559927 25808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:08.561013 25808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } }
I20260620 07:16:08.561362 25808 leader_election.cc:302] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18bf25cde111436c9df31ba6e63a5c7e; no voters:
I20260620 07:16:08.561743 25808 leader_election.cc:288] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:08.561851 25813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:08.562069 25813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [term 1 LEADER]: Becoming Leader. State: Replica: 18bf25cde111436c9df31ba6e63a5c7e, State: Running, Role: LEADER
I20260620 07:16:08.562419 25813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } }
I20260620 07:16:08.562897 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:08.563103 25808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:08.565223 25814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18bf25cde111436c9df31ba6e63a5c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } } }
I20260620 07:16:08.565354 25814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:08.565817 25815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18bf25cde111436c9df31ba6e63a5c7e. Latest consensus state: current_term: 1 leader_uuid: "18bf25cde111436c9df31ba6e63a5c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18bf25cde111436c9df31ba6e63a5c7e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41457 } } }
I20260620 07:16:08.566015 25815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e [sys.catalog]: This master's current role is: LEADER
W20260620 07:16:08.568275 25833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:08.568362 25833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:08.568466 25832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:08.569419 25832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:08.570724 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:08.565229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38487 (local address 127.3.245.62:41457)
0620 07:16:08.565342 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:08.565346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:08.566363 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 07:16:08.566520 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:08.566523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:08.566567 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:08.566655 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:08.566959 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:08.567714 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:08.568855 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:08.569172 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:08.569278 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:08.570238 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:08.570245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:08.570247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:08.570260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:08.570287 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:08.570300 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:08.570386 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:08.570559 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 07:16:08.570592 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:16:08.575030 25832 catalog_manager.cc:1347] Generated new cluster ID: 0674b82687d7430f9a4fce6a7d3fcc3b
I20260620 07:16:08.575095 25832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:08.585907 25832 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:08.587355 25832 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:08.592767 25832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18bf25cde111436c9df31ba6e63a5c7e: Generated new TSK 0
I20260620 07:16:08.593621 25832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:08.597214 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25740
W20260620 07:16:08.602632 25725 connection.cc:535] client connection to 127.3.245.62:41457 recv error: Network error: failed to read from TLS socket (remote: 127.3.245.62:41457): Connection reset by peer (error 104)
2026-06-20T07:16:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:08 dist-test-slave-ldfr krb5kdc[25839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:08 dist-test-slave-ldfr krb5kdc[25839](info): set up 2 sockets
Jun 20 07:16:08 dist-test-slave-ldfr krb5kdc[25839](info): commencing operation
krb5kdc: starting...
W20260620 07:16:09.652911 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:09 dist-test-slave-ldfr krb5kdc[25839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:09.775626 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:38723
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:09.888928 25852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:09.889271 25852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:09.889358 25852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:09.894501 25852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:09.894690 25852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:09.894742 25852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:09.894788 25852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:09.894825 25852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:09.901903 25852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:38723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:09.903193 25852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:09.904382 25852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:09.914536 25857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:09.914561 25860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:09.914536 25858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:09.915040 25852 server_base.cc:1034] running on GCE node
I20260620 07:16:09.915699 25852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:09.917133 25852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:09.918325 25852 hybrid_clock.cc:648] HybridClock initialized: now 1781939769918298 us; error 39 us; skew 500 ppm
Jun 20 07:16:09 dist-test-slave-ldfr krb5kdc[25839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:09.921401 25852 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:09.922329 25852 webserver.cc:460] Webserver started at http://127.3.245.62:43199/ using document root <none> and password file <none>
I20260620 07:16:09.922904 25852 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:09.922995 25852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:09.923267 25852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:09.925470 25852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "400df58059df4073a308a7f757c672e7"
format_stamp: "Formatted at 2026-06-20 07:16:09 on dist-test-slave-ldfr"
server_key: "a2a8be57ba77c3bb372cc575dc9cec1a"
server_key_iv: "3ac1dc6fbb54ac0db89d414788b7d09d"
server_key_version: "encryptionkey@0"
I20260620 07:16:09.926086 25852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "400df58059df4073a308a7f757c672e7"
format_stamp: "Formatted at 2026-06-20 07:16:09 on dist-test-slave-ldfr"
server_key: "a2a8be57ba77c3bb372cc575dc9cec1a"
server_key_iv: "3ac1dc6fbb54ac0db89d414788b7d09d"
server_key_version: "encryptionkey@0"
I20260620 07:16:09.929986 25852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:16:09.932540 25867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:09.933965 25852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:09.934128 25852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "400df58059df4073a308a7f757c672e7"
format_stamp: "Formatted at 2026-06-20 07:16:09 on dist-test-slave-ldfr"
server_key: "a2a8be57ba77c3bb372cc575dc9cec1a"
server_key_iv: "3ac1dc6fbb54ac0db89d414788b7d09d"
server_key_version: "encryptionkey@0"
I20260620 07:16:09.934250 25852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:09.938941 25852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:09.942112 25852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:09.942299 25852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:09.949072 25852 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:38723
I20260620 07:16:09.949280 25919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:38723 every 8 connection(s)
I20260620 07:16:09.950042 25852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:09.952044 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25852
I20260620 07:16:09.952241 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:09.952525 4052 external_mini_cluster.cc:1428] Setting key 8882947d905de9911d06ef5ff6b6c630
I20260620 07:16:09.952930 25920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:09.958319 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:09.954014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34289 (local address 127.3.245.62:38723)
0620 07:16:09.954439 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:09.954459 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:09.954503 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:16:09.955234 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:09.955257 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:09.955618 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:09.955998 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:09.956883 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:09.956889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:09.956906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:09.956922 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:09.956992 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:09.957018 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:09.957164 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:09.957267 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 07:16:09.957414 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260620 07:16:09.960202 25920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7: Bootstrap starting.
I20260620 07:16:09.961345 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:09.963024 25920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:09.963959 25920 log.cc:824] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:09.965723 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:09.963674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39159 (local address 127.3.245.62:38723)
0620 07:16:09.963813 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:09.963818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:09.964012 (+ 194us) server_negotiation.cc:407] Connection header received
0620 07:16:09.964149 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:09.964157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:09.964213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:09.964279 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:09.965164 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:09.965167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:09.965169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:09.965179 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:09.965224 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:09.965235 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:09.965302 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:09.965457 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 07:16:09.965559 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:16:09.966401 25920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7: No bootstrap required, opened a new log
I20260620 07:16:09.970255 25920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } }
I20260620 07:16:09.970487 25920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:09.970522 25920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400df58059df4073a308a7f757c672e7, State: Initialized, Role: FOLLOWER
I20260620 07:16:09.971005 25920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } }
I20260620 07:16:09.971125 25920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:09.971213 25920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:09.971311 25920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:09.972270 25920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } }
I20260620 07:16:09.972658 25920 leader_election.cc:302] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 400df58059df4073a308a7f757c672e7; no voters:
I20260620 07:16:09.973007 25920 leader_election.cc:288] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:09.973224 25930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:09.973558 25930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [term 1 LEADER]: Becoming Leader. State: Replica: 400df58059df4073a308a7f757c672e7, State: Running, Role: LEADER
I20260620 07:16:09.973865 25930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } }
I20260620 07:16:09.974310 25920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:09.975701 25931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "400df58059df4073a308a7f757c672e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } } }
I20260620 07:16:09.975744 25932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 400df58059df4073a308a7f757c672e7. Latest consensus state: current_term: 1 leader_uuid: "400df58059df4073a308a7f757c672e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df58059df4073a308a7f757c672e7" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 38723 } } }
I20260620 07:16:09.975837 25931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:09.975844 25932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:09.976286 25940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:09.979620 25940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:09.985733 25940 catalog_manager.cc:1347] Generated new cluster ID: 29a93598c2dc4844aef11fe7505653db
I20260620 07:16:09.985828 25940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:10.019243 25940 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:10.019616 25940 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:10.027011 25940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 400df58059df4073a308a7f757c672e7: Generated new TSK 0
I20260620 07:16:10.027730 25940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:10.036566 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25852
2026-06-20T07:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:10 dist-test-slave-ldfr krb5kdc[25951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:10 dist-test-slave-ldfr krb5kdc[25951](info): set up 2 sockets
Jun 20 07:16:10 dist-test-slave-ldfr krb5kdc[25951](info): commencing operation
krb5kdc: starting...
W20260620 07:16:11.072999 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[25951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:11.193639 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:32923
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:32923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:11.305938 25964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:11.306211 25964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:11.306254 25964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:11.310352 25964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:11.310415 25964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:11.310436 25964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:11.310452 25964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:11.310467 25964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:11.315822 25964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:32923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:11.317005 25964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:11.318045 25964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:11.326844 25969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:11.326815 25970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:11.326836 25972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:11.327086 25964 server_base.cc:1034] running on GCE node
I20260620 07:16:11.327764 25964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:11.328992 25964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:11.330197 25964 hybrid_clock.cc:648] HybridClock initialized: now 1781939771330163 us; error 46 us; skew 500 ppm
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[25951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939771, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:11.332955 25964 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:11.333875 25964 webserver.cc:460] Webserver started at http://127.3.245.62:45653/ using document root <none> and password file <none>
I20260620 07:16:11.334456 25964 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:11.334520 25964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:11.334693 25964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:11.336970 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "1ab34d96dfe749f881fce6d8f01a541e"
format_stamp: "Formatted at 2026-06-20 07:16:11 on dist-test-slave-ldfr"
server_key: "45c80c67dad178ca72a40bc50c8970ce"
server_key_iv: "5e9107f5721f1ec5e0268463ca3f471a"
server_key_version: "encryptionkey@0"
I20260620 07:16:11.337553 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "1ab34d96dfe749f881fce6d8f01a541e"
format_stamp: "Formatted at 2026-06-20 07:16:11 on dist-test-slave-ldfr"
server_key: "45c80c67dad178ca72a40bc50c8970ce"
server_key_iv: "5e9107f5721f1ec5e0268463ca3f471a"
server_key_version: "encryptionkey@0"
I20260620 07:16:11.341614 25964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:16:11.344058 25979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:11.345427 25964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:11.345573 25964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "1ab34d96dfe749f881fce6d8f01a541e"
format_stamp: "Formatted at 2026-06-20 07:16:11 on dist-test-slave-ldfr"
server_key: "45c80c67dad178ca72a40bc50c8970ce"
server_key_iv: "5e9107f5721f1ec5e0268463ca3f471a"
server_key_version: "encryptionkey@0"
I20260620 07:16:11.345682 25964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:11.372412 25964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:11.375487 25964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:11.375691 25964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:11.383035 25964 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:32923
I20260620 07:16:11.383060 26031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:32923 every 8 connection(s)
I20260620 07:16:11.383991 25964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:11.387008 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:11.390264 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 25964
I20260620 07:16:11.390375 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:11.390631 4052 external_mini_cluster.cc:1428] Setting key 6fe2264df0fb52e0588e21ef26a35ae4
I20260620 07:16:11.393678 26032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: Bootstrap starting.
I20260620 07:16:11.396847 26032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:11.397677 26032 log.cc:824] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:11.399921 26032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: No bootstrap required, opened a new log
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[25951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:11.404526 26032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } }
I20260620 07:16:11.404886 26032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:11.404966 26032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab34d96dfe749f881fce6d8f01a541e, State: Initialized, Role: FOLLOWER
I20260620 07:16:11.405503 26032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } }
I20260620 07:16:11.405712 26032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:11.405790 26032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:11.405894 26032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:11.407202 26032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } }
I20260620 07:16:11.407583 26032 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab34d96dfe749f881fce6d8f01a541e; no voters:
I20260620 07:16:11.407940 26032 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:11.408075 26037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:11.408252 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:11.392053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34851 (local address 127.3.245.62:32923)
0620 07:16:11.392527 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:11.392549 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:11.392596 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:16:11.393497 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:11.393522 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:11.394001 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:11.394385 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:11.395085 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:11.396839 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:11.397795 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:11.398471 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:11.398654 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:11.401547 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:11.401555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:11.401576 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:11.401612 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:11.404572 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:11.405141 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:11.405148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:11.405154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:11.405302 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:11.405578 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:11.405581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:11.405583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:11.406168 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:11.406372 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:11.406778 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 07:16:11.407142 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":127,"threads_started":1}
I20260620 07:16:11.408304 26037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [term 1 LEADER]: Becoming Leader. State: Replica: 1ab34d96dfe749f881fce6d8f01a541e, State: Running, Role: LEADER
I20260620 07:16:11.408675 26037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } }
I20260620 07:16:11.409302 26032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:11.410317 26039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab34d96dfe749f881fce6d8f01a541e. Latest consensus state: current_term: 1 leader_uuid: "1ab34d96dfe749f881fce6d8f01a541e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } } }
I20260620 07:16:11.410439 26039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:11.410807 26038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab34d96dfe749f881fce6d8f01a541e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab34d96dfe749f881fce6d8f01a541e" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 32923 } } }
I20260620 07:16:11.410907 26038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:11.411624 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 07:16:11.415308 26053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:11.415426 26053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:11.415499 26041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:11.416338 26041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:11.422097 26041 catalog_manager.cc:1347] Generated new cluster ID: c6592ae77f634769a15847cd6c1cdcf8
I20260620 07:16:11.422163 26041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:11.424700 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:11.414035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46565 (local address 127.3.245.62:32923)
0620 07:16:11.414163 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:11.414168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:11.414390 (+ 222us) server_negotiation.cc:407] Connection header received
0620 07:16:11.414544 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:11.414549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:11.414595 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:11.414698 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:11.415390 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:11.416066 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:11.417169 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:11.417462 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:11.417585 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:11.420965 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:11.420980 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:11.420983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:11.421016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:11.422866 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:11.423338 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:11.423344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:11.423346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:11.423436 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:11.423739 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:11.423744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:11.423746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:11.423941 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:11.424085 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:11.424343 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 07:16:11.424471 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:16:11.437743 26041 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:11.439016 26041 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:11.443943 26041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab34d96dfe749f881fce6d8f01a541e: Generated new TSK 0
I20260620 07:16:11.444741 26041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:11.449707 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 25964
2026-06-20T07:16:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[26063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[26063](info): set up 2 sockets
Jun 20 07:16:11 dist-test-slave-ldfr krb5kdc[26063](info): commencing operation
krb5kdc: starting...
W20260620 07:16:12.489034 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:12 dist-test-slave-ldfr krb5kdc[26063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:12.612922 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36347
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:12.724727 26076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:12.725005 26076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:12.725049 26076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:12.729208 26076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:12.729278 26076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:12.729297 26076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:12.729314 26076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:12.729370 26076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:12.734660 26076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:12.735725 26076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:12.736763 26076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:12.745338 26082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:12.745414 26076 server_base.cc:1034] running on GCE node
W20260620 07:16:12.745317 26081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:12.745358 26084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:12.746287 26076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:12.747449 26076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:12.748813 26076 hybrid_clock.cc:648] HybridClock initialized: now 1781939772748779 us; error 46 us; skew 500 ppm
Jun 20 07:16:12 dist-test-slave-ldfr krb5kdc[26063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939772, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:12.751504 26076 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:12.752324 26076 webserver.cc:460] Webserver started at http://127.3.245.62:36025/ using document root <none> and password file <none>
I20260620 07:16:12.752908 26076 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:12.752971 26076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:12.753136 26076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:12.755064 26076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "09cfacd549a84b3387a8945e7c295f64"
format_stamp: "Formatted at 2026-06-20 07:16:12 on dist-test-slave-ldfr"
server_key: "2efb928d63d4ec57372f7079d8821ab8"
server_key_iv: "762ff6fd954c9c9bf71e5ee07f6b8459"
server_key_version: "encryptionkey@0"
I20260620 07:16:12.755558 26076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "09cfacd549a84b3387a8945e7c295f64"
format_stamp: "Formatted at 2026-06-20 07:16:12 on dist-test-slave-ldfr"
server_key: "2efb928d63d4ec57372f7079d8821ab8"
server_key_iv: "762ff6fd954c9c9bf71e5ee07f6b8459"
server_key_version: "encryptionkey@0"
I20260620 07:16:12.759325 26076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 07:16:12.761818 26091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:12.763085 26076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:12.763226 26076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "09cfacd549a84b3387a8945e7c295f64"
format_stamp: "Formatted at 2026-06-20 07:16:12 on dist-test-slave-ldfr"
server_key: "2efb928d63d4ec57372f7079d8821ab8"
server_key_iv: "762ff6fd954c9c9bf71e5ee07f6b8459"
server_key_version: "encryptionkey@0"
I20260620 07:16:12.763334 26076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:12.781579 26076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:12.785164 26076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:12.785388 26076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:12.792900 26076 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:36347
I20260620 07:16:12.792961 26143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:36347 every 8 connection(s)
I20260620 07:16:12.793910 26076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:12.796733 26144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:12.799328 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26076
I20260620 07:16:12.799515 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:12.799856 4052 external_mini_cluster.cc:1428] Setting key 04d1b8a749fec67d1d055a53f2a83092
I20260620 07:16:12.803064 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64: Bootstrap starting.
I20260620 07:16:12.805771 26144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:12.806685 26144 log.cc:824] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:12.808979 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64: No bootstrap required, opened a new log
I20260620 07:16:12.809249 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:12.801282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54987 (local address 127.3.245.62:36347)
0620 07:16:12.801715 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:12.801735 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:12.801775 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:16:12.802412 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:12.802430 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:12.802780 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:12.803064 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:12.803480 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:12.804434 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:12.805675 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:12.806232 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:12.806358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:12.807299 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:12.807307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:12.807328 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:12.807809 (+ 481us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:12.807863 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:12.807887 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:12.807964 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:12.808143 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:16:12.808359 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260620 07:16:12.812050 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:12.812861 26144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } }
I20260620 07:16:12.813129 26144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:12.813232 26144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09cfacd549a84b3387a8945e7c295f64, State: Initialized, Role: FOLLOWER
I20260620 07:16:12.813727 26144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } }
I20260620 07:16:12.813863 26144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:12.813951 26144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:12.814081 26144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:12.815407 26144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } }
I20260620 07:16:12.815819 26144 leader_election.cc:302] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09cfacd549a84b3387a8945e7c295f64; no voters:
I20260620 07:16:12.816201 26144 leader_election.cc:288] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:12.816288 26154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:12.816524 26154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [term 1 LEADER]: Becoming Leader. State: Replica: 09cfacd549a84b3387a8945e7c295f64, State: Running, Role: LEADER
I20260620 07:16:12.816907 26154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } }
I20260620 07:16:12.817668 26144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:12.818783 26155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09cfacd549a84b3387a8945e7c295f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } } }
I20260620 07:16:12.818902 26155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:12.818753 26156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09cfacd549a84b3387a8945e7c295f64. Latest consensus state: current_term: 1 leader_uuid: "09cfacd549a84b3387a8945e7c295f64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09cfacd549a84b3387a8945e7c295f64" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 36347 } } }
I20260620 07:16:12.819226 26156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:12.819787 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:12.814290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38135 (local address 127.3.245.62:36347)
0620 07:16:12.814427 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:12.814431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:12.814598 (+ 167us) server_negotiation.cc:407] Connection header received
0620 07:16:12.814715 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:12.814717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:12.814756 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:12.814818 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:12.815118 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:12.815710 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:12.816859 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:12.817185 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:12.817264 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:12.819206 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:12.819210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:12.819212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:12.819226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:12.819256 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:12.819270 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:12.819320 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:12.819530 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 07:16:12.819608 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:16:12.820284 26163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:12.823035 26163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:12.829560 26163 catalog_manager.cc:1347] Generated new cluster ID: 766a7cb3f191466caa754fa8237b8f39
I20260620 07:16:12.829663 26163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:12.871608 26163 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:12.872568 26163 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:12.878199 26163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09cfacd549a84b3387a8945e7c295f64: Generated new TSK 0
I20260620 07:16:12.879040 26163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:12.892263 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26076
2026-06-20T07:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:12 dist-test-slave-ldfr krb5kdc[26175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:12 dist-test-slave-ldfr krb5kdc[26175](info): set up 2 sockets
Jun 20 07:16:12 dist-test-slave-ldfr krb5kdc[26175](info): commencing operation
krb5kdc: starting...
W20260620 07:16:13.933375 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:14 dist-test-slave-ldfr krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:14.055917 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35399
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:35399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:14.167266 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:14.167582 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:14.167665 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:14.171785 26188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:14.171883 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:14.171921 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:14.171979 26188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:14.172014 26188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:14.177481 26188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:35399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:35399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:14.178586 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:14.179633 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:14.188211 26193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:14.188254 26196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:14.188267 26194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:14.188437 26188 server_base.cc:1034] running on GCE node
I20260620 07:16:14.189142 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:14.190274 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:14.191471 26188 hybrid_clock.cc:648] HybridClock initialized: now 1781939774191484 us; error 58 us; skew 500 ppm
Jun 20 07:16:14 dist-test-slave-ldfr krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939774, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:14.194474 26188 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:14.195432 26188 webserver.cc:460] Webserver started at http://127.3.245.62:34423/ using document root <none> and password file <none>
I20260620 07:16:14.196015 26188 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:14.196102 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:14.196329 26188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:14.198436 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "840c07e96579437c859d7416606b35b4"
format_stamp: "Formatted at 2026-06-20 07:16:14 on dist-test-slave-ldfr"
server_key: "46ad112df04a78c3133029fdc569ee28"
server_key_iv: "50fe80112d85ef7fd9e4a4873b1682d6"
server_key_version: "encryptionkey@0"
I20260620 07:16:14.199007 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "840c07e96579437c859d7416606b35b4"
format_stamp: "Formatted at 2026-06-20 07:16:14 on dist-test-slave-ldfr"
server_key: "46ad112df04a78c3133029fdc569ee28"
server_key_iv: "50fe80112d85ef7fd9e4a4873b1682d6"
server_key_version: "encryptionkey@0"
I20260620 07:16:14.202710 26188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 07:16:14.205139 26203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:14.206365 26188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 07:16:14.206503 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "840c07e96579437c859d7416606b35b4"
format_stamp: "Formatted at 2026-06-20 07:16:14 on dist-test-slave-ldfr"
server_key: "46ad112df04a78c3133029fdc569ee28"
server_key_iv: "50fe80112d85ef7fd9e4a4873b1682d6"
server_key_version: "encryptionkey@0"
I20260620 07:16:14.206609 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:14.211141 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:14.214293 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:14.214493 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:14.221510 26188 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:35399
I20260620 07:16:14.221529 26255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:35399 every 8 connection(s)
I20260620 07:16:14.222497 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:14.225265 26256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:14.230646 26256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: Bootstrap starting.
I20260620 07:16:14.232395 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26188
I20260620 07:16:14.232580 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:14.232900 4052 external_mini_cluster.cc:1428] Setting key 6c873b07da6052e9391a03d7ef43c402
I20260620 07:16:14.233237 26256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:14.234122 26256 log.cc:824] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:14.236171 26256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: No bootstrap required, opened a new log
I20260620 07:16:14.238011 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:14.234156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40641 (local address 127.3.245.62:35399)
0620 07:16:14.234618 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:14.234633 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:14.234668 (+ 35us) server_negotiation.cc:407] Connection header received
0620 07:16:14.235303 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:14.235319 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:14.235581 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:14.235854 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:14.236775 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:14.236780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:14.236792 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:14.236803 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:14.236859 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:14.236879 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:14.236974 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:14.237129 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 07:16:14.237303 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":150,"threads_started":1}
I20260620 07:16:14.240283 26256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } }
I20260620 07:16:14.240520 26256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:16:14.240607 26256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840c07e96579437c859d7416606b35b4, State: Initialized, Role: FOLLOWER
I20260620 07:16:14.241132 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:14.241150 26256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } }
I20260620 07:16:14.241304 26256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:16:14.241364 26256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:16:14.241454 26256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:14.242686 26256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } }
I20260620 07:16:14.243060 26256 leader_election.cc:302] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 840c07e96579437c859d7416606b35b4; no voters:
I20260620 07:16:14.243450 26256 leader_election.cc:288] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:16:14.243516 26265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:14.243762 26265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [term 1 LEADER]: Becoming Leader. State: Replica: 840c07e96579437c859d7416606b35b4, State: Running, Role: LEADER
I20260620 07:16:14.244058 26265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } }
I20260620 07:16:14.244846 26256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:14.245287 26268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "840c07e96579437c859d7416606b35b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } } }
I20260620 07:16:14.245425 26268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:14.245569 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:14.243509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57789 (local address 127.3.245.62:35399)
0620 07:16:14.243627 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:14.243632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:14.243885 (+ 253us) server_negotiation.cc:407] Connection header received
0620 07:16:14.244120 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:14.244124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:14.244161 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:14.244244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:14.245097 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:14.245102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:14.245104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:16:14.245117 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:14.245145 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:16:14.245158 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:14.245232 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:14.245397 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 07:16:14.245441 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:16:14.245702 26267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 840c07e96579437c859d7416606b35b4. Latest consensus state: current_term: 1 leader_uuid: "840c07e96579437c859d7416606b35b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "840c07e96579437c859d7416606b35b4" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 35399 } } }
I20260620 07:16:14.245823 26267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4 [sys.catalog]: This master's current role is: LEADER
W20260620 07:16:14.249639 26281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:14.249734 26281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:14.249807 26271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:14.250480 26271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:14.256047 26271 catalog_manager.cc:1347] Generated new cluster ID: 868e9a094f8b48c98911b006d5a85b04
I20260620 07:16:14.256125 26271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:14.276825 26271 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:14.277208 26271 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:14.286722 26271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 840c07e96579437c859d7416606b35b4: Generated new TSK 0
I20260620 07:16:14.287601 26271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:16:14.294970 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26188
2026-06-20T07:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1401 ms)
[----------] 18 tests from AuthTokenIssuingTest (25866 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:14 dist-test-slave-ldfr krb5kdc[26287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:14 dist-test-slave-ldfr krb5kdc[26287](info): set up 2 sockets
Jun 20 07:16:14 dist-test-slave-ldfr krb5kdc[26287](info): commencing operation
krb5kdc: starting...
W20260620 07:16:15.361105 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:15.483816 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41105
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:15.607116 26300 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:15.607403 26300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:15.607429 26300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:15.607467 26300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:15.611591 26300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:15.611653 26300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:15.611680 26300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:15.611696 26300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:15.611711 26300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:15.617105 26300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:15.618134 26300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:15.619109 26300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:15.627842 26308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:15.627842 26305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:15.627977 26300 server_base.cc:1034] running on GCE node
W20260620 07:16:15.628160 26306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:15.628834 26300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:15.630071 26300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:15.631238 26300 hybrid_clock.cc:648] HybridClock initialized: now 1781939775631209 us; error 44 us; skew 500 ppm
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:15.640755 26300 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:15.641800 26300 webserver.cc:460] Webserver started at http://127.3.245.62:33875/ using document root <none> and password file <none>
I20260620 07:16:15.642434 26300 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:15.642499 26300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:15.642724 26300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:15.644938 26300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "00f7d7762e5d4f7f976683e00a49050f"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "be55aa3612e05f6fbeaf7d4b774909ef"
server_key_iv: "3dcaac47660f1a036902163fef765034"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.645519 26300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "00f7d7762e5d4f7f976683e00a49050f"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "be55aa3612e05f6fbeaf7d4b774909ef"
server_key_iv: "3dcaac47660f1a036902163fef765034"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.649480 26300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:16:15.652058 26315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:15.653632 26300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:16:15.653792 26300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "00f7d7762e5d4f7f976683e00a49050f"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "be55aa3612e05f6fbeaf7d4b774909ef"
server_key_iv: "3dcaac47660f1a036902163fef765034"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.653934 26300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:15.664431 26300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:15.667481 26300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:15.667704 26300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:15.675186 26300 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:41105
I20260620 07:16:15.675567 26367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:41105 every 8 connection(s)
I20260620 07:16:15.676218 26300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:15.679023 26368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:15.680876 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26300
I20260620 07:16:15.681042 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:15.681310 4052 external_mini_cluster.cc:1428] Setting key 947f801c38ca7545948557615d6323c5
I20260620 07:16:15.682999 26368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.245.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.61@KRBTEST.COM" created.
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:15.700418 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.685562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41105 (local address 127.0.0.1:53092)
0620 07:16:15.686046 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.686098 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:15.686959 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:15.687814 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:15.687818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:15.688148 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:15.688491 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.688503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.690193 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.690196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.691290 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.691302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.692321 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.692328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.692466 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.693119 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:15.693145 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:15.694831 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:15.697379 (+ 2548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.697387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.697396 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.697629 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.698065 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.698067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.698069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.698176 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.698995 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:15.699000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:15.699149 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 07:16:15.699295 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:16:15.699611 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":39,"thread_start_us":131,"threads_started":1}
I20260620 07:16:15.700424 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.685562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.3.245.62:41105)
0620 07:16:15.686185 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:15.686200 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:15.686251 (+ 51us) server_negotiation.cc:407] Connection header received
0620 07:16:15.687307 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:15.687315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:15.687569 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:15.687814 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:15.688699 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.690030 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.691477 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.692111 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.692298 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.694977 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:15.694987 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:15.695004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:15.695036 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:15.697250 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.697779 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.697788 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.697792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.697952 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.698277 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.698284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.698287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.698864 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:15.699008 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:15.699419 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 07:16:15.699719 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":485,"thread_start_us":168,"threads_started":1}
I20260620 07:16:15.702896 26368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:15.703717 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.703391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47230)
0620 07:16:15.703544 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.703617 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:16:15.704066 26318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
I20260620 07:16:15.705109 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.704847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47244)
0620 07:16:15.704991 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.705047 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:16:15.705304 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43443: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
I20260620 07:16:15.708949 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43443
--webserver_interface=127.3.245.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:16:15.754560 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } attempt: 1
I20260620 07:16:15.755463 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.755163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47246)
0620 07:16:15.755320 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.755386 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 07:16:15.756186 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.755976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47256)
0620 07:16:15.756081 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.756131 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:16:15.756311 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43443: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
W20260620 07:16:15.820030 26375 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:15.820398 26375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:15.820474 26375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:15.820570 26375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:16:15.824473 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } attempt: 2
W20260620 07:16:15.824792 26375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:15.824869 26375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:15.824913 26375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:15.824949 26375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:15.824981 26375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:15.825359 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.825055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47260)
0620 07:16:15.825217 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.825284 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:16:15.826074 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.825856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47262)
0620 07:16:15.825964 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.826017 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:16:15.826198 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43443: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43443: connect: Connection refused (error 111)
I20260620 07:16:15.830529 26375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.245.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:15.831650 26375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:15.832703 26375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:15.841046 26375 server_base.cc:1034] running on GCE node
W20260620 07:16:15.840960 26380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:15.840972 26381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:15.840960 26383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:15.841768 26375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:15.842897 26375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:15.844131 26375 hybrid_clock.cc:648] HybridClock initialized: now 1781939775844098 us; error 27 us; skew 500 ppm
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:15.846889 26375 init.cc:377] Logged in from keytab as kudu/127.3.245.61@KRBTEST.COM (short username kudu)
I20260620 07:16:15.847777 26375 webserver.cc:460] Webserver started at http://127.3.245.61:41597/ using document root <none> and password file <none>
I20260620 07:16:15.848294 26375 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:15.848349 26375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:15.848508 26375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:15.850512 26375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data/instance:
uuid: "b3489711028c4bd7a26100876df55d05"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "b6e297692cc650a20d4473e0fe73e1c2"
server_key_iv: "a52eb41628f2ec288cd1b181f2cf1f3f"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.851009 26375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal/instance:
uuid: "b3489711028c4bd7a26100876df55d05"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "b6e297692cc650a20d4473e0fe73e1c2"
server_key_iv: "a52eb41628f2ec288cd1b181f2cf1f3f"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.854566 26375 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 07:16:15.856848 26390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:15.858130 26375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:16:15.858227 26375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "b3489711028c4bd7a26100876df55d05"
format_stamp: "Formatted at 2026-06-20 07:16:15 on dist-test-slave-ldfr"
server_key: "b6e297692cc650a20d4473e0fe73e1c2"
server_key_iv: "a52eb41628f2ec288cd1b181f2cf1f3f"
server_key_version: "encryptionkey@0"
I20260620 07:16:15.858309 26375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:15.890141 26375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:15.893952 26375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:15.894199 26375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:15.902185 26375 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.61:43443
I20260620 07:16:15.902529 26442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.61:43443 every 8 connection(s)
I20260620 07:16:15.903331 26375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
I20260620 07:16:15.905547 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26375
I20260620 07:16:15.905678 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-1/wal/instance
I20260620 07:16:15.905952 4052 external_mini_cluster.cc:1428] Setting key 9cc8bd4306ec7a88276e59cad459cbe8
I20260620 07:16:15.906711 26443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:15.911016 26443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.245.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.60@KRBTEST.COM" created.
I20260620 07:16:15.917354 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } attempt: 3
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:15.927899 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.917844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47270)
0620 07:16:15.918038 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.918055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:15.918167 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:15.918796 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:15.918799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:15.919017 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:15.919141 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.919148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.920365 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.920370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.921122 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.921132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.921724 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.921727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.921834 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.922295 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:15.922309 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:15.923644 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:15.926123 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.926135 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.926410 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.926720 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.926727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.926819 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.927340 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:15.927343 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:15.927462 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 07:16:15.927551 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 07:16:15.927765 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"spinlock_wait_cycles":38144}
I20260620 07:16:15.928447 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.913094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.3.245.62:41105)
0620 07:16:15.913293 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:15.913298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:15.914193 (+ 895us) server_negotiation.cc:407] Connection header received
0620 07:16:15.915280 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:15.915284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:15.915333 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:15.915431 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:15.916992 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.917680 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.919478 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.919845 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.919953 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.923244 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:15.923248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:15.923251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:15.923278 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:15.925289 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926006 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.926010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.926012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.926100 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926446 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.926449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.926451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.926705 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:15.926798 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:15.928196 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0620 07:16:15.928310 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:16:15.929145 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.913524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41105 (local address 127.0.0.1:53098)
0620 07:16:15.914024 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.914076 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:15.915006 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:15.915702 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:15.915709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:15.916077 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:15.916756 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.916773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.917841 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.917847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.919343 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.919358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.920014 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.920022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.920266 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.920998 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:15.921024 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:15.923061 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:15.925421 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.925434 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.925450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.925892 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.926180 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.926186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.926328 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.926801 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:15.926810 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:15.927175 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 07:16:15.928305 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0620 07:16:15.928536 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":140,"threads_started":1}
I20260620 07:16:15.929154 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.917969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.3.245.61:43443)
0620 07:16:15.918398 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:15.918407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:15.918427 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:16:15.918462 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:15.918466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:15.918627 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:15.918812 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:15.919263 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.920259 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.921244 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.921591 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.921723 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.923839 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:15.923846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:15.923863 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:15.923892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:15.926006 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926517 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.926522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.926524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.926607 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.926924 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.926928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.926929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.927215 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:15.927355 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:15.927705 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:16:15.928098 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":182,"threads_started":1}
I20260620 07:16:15.930543 26443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:15.931229 26368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:15.931893 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.931658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53638)
0620 07:16:15.931777 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.931827 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:16:15.932547 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.932316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53644)
0620 07:16:15.932444 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.932489 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 07:16:15.932700 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
I20260620 07:16:15.935322 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:44405
--webserver_interface=127.3.245.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:46765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:16:15 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:15.939289 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.931081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47278)
0620 07:16:15.931316 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.931331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:15.931407 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:15.931644 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:15.931646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:15.931827 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:15.931939 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.931949 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.932838 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.932840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.933460 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:15.933468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.933969 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.933972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:15.934054 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.934560 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:15.934579 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:15.935860 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:15.937907 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.937909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.937911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.938097 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.938380 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:15.938382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:15.938383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:15.938455 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:15.938897 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:15.938899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:15.938973 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 07:16:15.939053 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:16:15.939150 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 07:16:15.939545 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.931139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.3.245.61:43443)
0620 07:16:15.931449 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:15.931453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:15.931466 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:15.931507 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:15.931509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:15.931543 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:15.931634 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:15.932064 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.932717 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.933560 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:15.933846 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:15.933932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:15.935987 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:15.935991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:15.935996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:15.936021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:15.937781 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.938190 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.938196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.938201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.938287 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:15.938542 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:15.938548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:15.938553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:15.938805 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:15.938905 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:15.939139 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 07:16:15.939272 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
I20260620 07:16:15.940209 26443 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:15.941030 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.940694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53652)
0620 07:16:15.940842 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.940949 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:16:15.941605 26393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
I20260620 07:16:15.942544 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.942265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53658)
0620 07:16:15.942403 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.942469 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:16:15.942734 26443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
I20260620 07:16:15.963861 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 1
I20260620 07:16:15.964784 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.964420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53668)
0620 07:16:15.964572 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.964704 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:16:15.965487 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.965274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53672)
0620 07:16:15.965384 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.965435 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:16:15.965616 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
I20260620 07:16:15.992038 26443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 1
I20260620 07:16:15.993009 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.992648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53686)
0620 07:16:15.992841 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.992915 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 07:16:15.993799 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:15.993567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53694)
0620 07:16:15.993681 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:15.993735 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:16:15.993945 26443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
I20260620 07:16:16.040836 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 2
I20260620 07:16:16.041694 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.041405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53708)
0620 07:16:16.041547 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.041615 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:16:16.042644 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.042378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53716)
0620 07:16:16.042500 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.042558 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:16:16.042830 26368 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
W20260620 07:16:16.049099 26450 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:16.049376 26450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:16.049455 26450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:16.049504 26450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:16.053668 26450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:16.053735 26450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:16.053754 26450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:16.053772 26450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:16.053823 26450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:16.059458 26450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:46765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:41105,127.3.245.61:43443,127.3.245.60:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.245.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:16.060586 26450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:16.061673 26450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:16.062162 26443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 2
I20260620 07:16:16.063133 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.062803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53726)
0620 07:16:16.062963 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.063035 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:16:16.064040 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.063796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53738)
0620 07:16:16.063913 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.063971 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:16:16.064213 26443 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:44405: Network error: Client connection negotiation failed: client connection to 127.3.245.60:44405: connect: Connection refused (error 111)
W20260620 07:16:16.071290 26457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:16.071305 26456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:16.071360 26459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:16.071563 26450 server_base.cc:1034] running on GCE node
I20260620 07:16:16.072206 26450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:16.073503 26450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:16.074730 26450 hybrid_clock.cc:648] HybridClock initialized: now 1781939776074711 us; error 34 us; skew 500 ppm
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:16.077603 26450 init.cc:377] Logged in from keytab as kudu/127.3.245.60@KRBTEST.COM (short username kudu)
I20260620 07:16:16.078572 26450 webserver.cc:460] Webserver started at http://127.3.245.60:43559/ using document root <none> and password file <none>
I20260620 07:16:16.079119 26450 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:16.079180 26450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:16.079351 26450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:16.081485 26450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data/instance:
uuid: "4a8e40ab4b23450989480630de097df2"
format_stamp: "Formatted at 2026-06-20 07:16:16 on dist-test-slave-ldfr"
server_key: "316ad1f3f98a1d11c4611c0d0901140e"
server_key_iv: "6de20dd5d5d0d42611cb59a94d7b6a1f"
server_key_version: "encryptionkey@0"
I20260620 07:16:16.082041 26450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal/instance:
uuid: "4a8e40ab4b23450989480630de097df2"
format_stamp: "Formatted at 2026-06-20 07:16:16 on dist-test-slave-ldfr"
server_key: "316ad1f3f98a1d11c4611c0d0901140e"
server_key_iv: "6de20dd5d5d0d42611cb59a94d7b6a1f"
server_key_version: "encryptionkey@0"
I20260620 07:16:16.085899 26450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 07:16:16.088451 26466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:16.089910 26450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:16:16.090039 26450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "4a8e40ab4b23450989480630de097df2"
format_stamp: "Formatted at 2026-06-20 07:16:16 on dist-test-slave-ldfr"
server_key: "316ad1f3f98a1d11c4611c0d0901140e"
server_key_iv: "6de20dd5d5d0d42611cb59a94d7b6a1f"
server_key_version: "encryptionkey@0"
I20260620 07:16:16.090154 26450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:16.115837 26450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:16.119278 26450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:16.119447 26450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:16.126691 26450 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.60:44405
I20260620 07:16:16.126989 26518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.60:44405 every 8 connection(s)
I20260620 07:16:16.127713 26450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
I20260620 07:16:16.130827 26519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:16.132956 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26450
I20260620 07:16:16.133081 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781939488540940-4052-0/minicluster-data/master-2/wal/instance
I20260620 07:16:16.133390 4052 external_mini_cluster.cc:1428] Setting key 1b40fbd9d3a0373bee4b3627232b3e24
I20260620 07:16:16.135574 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:16.142000 26368 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 3
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:16:16.148989 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.137790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.3.245.62:41105)
0620 07:16:16.137920 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.137925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.138209 (+ 284us) server_negotiation.cc:407] Connection header received
0620 07:16:16.138321 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.138326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.138389 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.138501 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.139277 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.140096 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.141271 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.141566 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.141692 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.144581 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.144585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.144588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.144635 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.146404 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.147024 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.147028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.147030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.147130 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.147569 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.147573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.147575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.147828 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.147923 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.148664 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.148803 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":768}
I20260620 07:16:16.150632 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.137476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:41105 (local address 127.0.0.1:53102)
0620 07:16:16.138043 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.138094 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:16.138222 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:16.138570 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:16.138579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:16.138952 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:16.139121 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.139131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.140234 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.140237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.141147 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.141158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.141752 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.141754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.141900 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.142545 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:16.142566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:16.144408 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:16.146520 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.146528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.146540 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.146893 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.147256 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.147259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.147261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.147470 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.147917 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:16.147924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:16.148318 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 07:16:16.149189 (+ 871us) client_negotiation.cc:240] Negotiation successful
0620 07:16:16.149543 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
I20260620 07:16:16.150632 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.134906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.3.245.60:44405)
0620 07:16:16.135420 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.135435 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.135476 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:16:16.136429 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.136452 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.136898 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.137255 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.138403 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.139428 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.140580 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.141297 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.141419 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.145068 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.145075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.145090 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.145118 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.147131 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.147689 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.147696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.147698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.147773 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.148320 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.148323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.148325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.148635 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.148844 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.149397 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.149612 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":366,"thread_start_us":152,"threads_started":1}
I20260620 07:16:16.151998 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:16.152268 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.142437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53748)
0620 07:16:16.142542 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.142558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:16.142636 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:16.143113 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:16.143119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:16.143323 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:16.143400 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.143405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.144144 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.144148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.144776 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.144785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.145779 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.145785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.145893 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.146367 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:16.146395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:16.147640 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:16.150380 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.150383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.150386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.150582 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.151194 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.151197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.151199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.151290 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.151802 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:16.151805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:16.151902 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 07:16:16.151998 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:16:16.152126 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"spinlock_wait_cycles":2944}
I20260620 07:16:16.153188 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.142516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.3.245.60:44405)
0620 07:16:16.142858 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.142868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.142881 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:16.142915 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.142920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.142965 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.143065 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.143514 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.144044 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.144882 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.145265 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.145386 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.147756 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.147760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.147763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.147786 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.150266 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.150697 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.150701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.150702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.150789 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.151551 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.151553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.151554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.151707 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.151811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.152573 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.152864 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260620 07:16:16.156466 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: Bootstrap starting.
I20260620 07:16:16.156666 26443 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } attempt: 3
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:16.159798 26368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:16.162009 26368 log.cc:824] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:16.162412 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.152837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.3.245.61:43443)
0620 07:16:16.152968 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.152972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.153048 (+ 76us) server_negotiation.cc:407] Connection header received
0620 07:16:16.153157 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.153162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.153224 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.153306 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.154036 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.154842 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.155643 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.155906 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.156015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.158390 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.158396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.158398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.158422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.160512 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.160991 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.160994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.160996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.161078 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.161413 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.161417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.161418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.161635 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.161717 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.162105 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.162231 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:16:16.162887 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.152596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43443 (local address 127.0.0.1:47286)
0620 07:16:16.152964 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.152979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:16.153070 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:16.153589 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:16.153592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:16.153769 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:16.153889 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.153899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.154984 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.154987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.155527 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.155534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.156022 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.156026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.156120 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.156676 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:16.156695 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:16.158256 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:16.160646 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.160649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.160652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.160889 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.161209 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.161214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.161219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.161310 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.161774 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:16.161776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:16.161874 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 07:16:16.162589 (+ 715us) client_negotiation.cc:240] Negotiation successful
0620 07:16:16.162729 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260620 07:16:16.163780 26519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:16.164707 26368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: No bootstrap required, opened a new log
I20260620 07:16:16.165830 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.153548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.3.245.62:41105)
0620 07:16:16.153937 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.153943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.153957 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:16:16.153988 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.153990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.154033 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.154110 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.154840 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.155572 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.156647 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.156931 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.157004 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.160445 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.160451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.160455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.160521 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.162650 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.164345 (+ 1695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.164349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.164351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.164441 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.164785 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.164788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.164790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.164989 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.165132 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.165485 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.165620 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305}
I20260620 07:16:16.166633 26446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.157039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53752)
0620 07:16:16.157279 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.157295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:16.157369 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:16.157935 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:16.157940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:16.158115 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:16.158223 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.158229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.159392 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.159398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.160268 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.160282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.160982 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.160986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.161086 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.161465 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:16.161478 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:16.162928 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:16.165045 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.165050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.165052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.165297 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.165561 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.165564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.165566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.165652 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.166103 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:16.166106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:16.166212 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:16:16.166335 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 07:16:16.166467 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 07:16:16.167212 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.157536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.3.245.60:44405)
0620 07:16:16.157668 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.157672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.157684 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:16:16.157715 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.157718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.157755 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.158331 (+ 576us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.158414 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.159260 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.160428 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.160845 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.161331 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.163075 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.163081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.163085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.163110 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.164941 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.165379 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.165382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.165383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.165459 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.165770 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.165773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.165774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.165985 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.166093 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.166931 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.167051 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:16:16.169097 26368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.169342 26368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f7d7762e5d4f7f976683e00a49050f, State: Initialized, Role: FOLLOWER
I20260620 07:16:16.169768 26368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.170362 26443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: Bootstrap starting.
W20260620 07:16:16.171279 26321 tablet.cc:2369] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:16:16.171274 26368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
Jun 20 07:16:16 dist-test-slave-ldfr krb5kdc[26287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:16.173983 26443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:16.175215 26443 log.cc:824] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:16.176707 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.164456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:44405 (local address 127.0.0.1:53768)
0620 07:16:16.165129 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:16.165142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:16.165218 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:16.168066 (+ 2848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:16.168068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:16.168249 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:16.168333 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.168338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.169281 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.169284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.170048 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:16.170062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.170662 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.170665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:16.170733 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.171343 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:16.171369 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:16.173151 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:16.175206 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.175209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.175211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.175435 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.175698 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:16.175701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:16.175703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:16.175778 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:16.176207 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:16.176209 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:16.176290 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 07:16:16.176382 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:16:16.176514 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592}
I20260620 07:16:16.177419 26443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: No bootstrap required, opened a new log
W20260620 07:16:16.177901 26537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:16.177987 26537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:16.178295 26526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.178426 26526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.178916 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.164701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.3.245.60:44405)
0620 07:16:16.167804 (+ 3103us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.167808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.167820 (+ 12us) server_negotiation.cc:407] Connection header received
0620 07:16:16.167853 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.167856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.167894 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.167962 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.168440 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.169153 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.170188 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.170480 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.170568 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.173314 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.173317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.173321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.173343 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.175076 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.175535 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.175537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.175539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.175612 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.175861 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.175864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.175865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.176066 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.176162 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.178655 (+ 2493us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.178782 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3049}
I20260620 07:16:16.180554 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.167168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.3.245.61:43443)
0620 07:16:16.167295 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.167301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.167612 (+ 311us) server_negotiation.cc:407] Connection header received
0620 07:16:16.167708 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.167711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.167758 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.167829 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.168708 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.169401 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.170733 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.171069 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.171188 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.176653 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.176657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.176660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.176686 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.178388 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.179245 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.179248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.179250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.179343 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.179657 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.179660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.179662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.179888 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.180080 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.180295 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.180419 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:16:16.182965 26443 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.183228 26443 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3489711028c4bd7a26100876df55d05, State: Initialized, Role: FOLLOWER
I20260620 07:16:16.183559 26443 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.184072 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.184140 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: Bootstrap starting.
I20260620 07:16:16.184288 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.184957 26443 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:16.186955 26519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:16.190692 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 07:16:16.190963 26551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:16.191084 26551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:16.187289 26300 master_runner.cc:186] Error getting master registration for 127.3.245.61:43443: OK, instance_id { permanent_uuid: "b3489711028c4bd7a26100876df55d05" instance_seqno: 1781939775899946 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:16:16.191422 26300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:16:16.191996 26341 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "00f7d7762e5d4f7f976683e00a49050f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53104
I20260620 07:16:16.192214 26341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:16:16.192360 26341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:16.192957 26519 log.cc:824] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:16.193827 26341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.194761 26341 leader_election.cc:288] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [CANDIDATE]: Term 1 election: Requested vote from peers b3489711028c4bd7a26100876df55d05 (127.3.245.61:43443), 4a8e40ab4b23450989480630de097df2 (127.3.245.60:44405)
I20260620 07:16:16.195327 26519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: No bootstrap required, opened a new log
I20260620 07:16:16.195569 26417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00f7d7762e5d4f7f976683e00a49050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b3489711028c4bd7a26100876df55d05"
I20260620 07:16:16.195808 26417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:16.197343 26417 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f7d7762e5d4f7f976683e00a49050f in term 1.
I20260620 07:16:16.197773 26318 leader_election.cc:302] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f7d7762e5d4f7f976683e00a49050f, b3489711028c4bd7a26100876df55d05; no voters:
I20260620 07:16:16.198088 26526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:16.198287 26526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [term 1 LEADER]: Becoming Leader. State: Replica: 00f7d7762e5d4f7f976683e00a49050f, State: Running, Role: LEADER
I20260620 07:16:16.198613 26526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.199770 26519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.200011 26519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8e40ab4b23450989480630de097df2, State: Initialized, Role: FOLLOWER
I20260620 07:16:16.200331 26519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } }
I20260620 07:16:16.200453 26526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f7d7762e5d4f7f976683e00a49050f. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.200567 26526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:16.203238 26493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00f7d7762e5d4f7f976683e00a49050f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4a8e40ab4b23450989480630de097df2"
I20260620 07:16:16.203816 26493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:16.205865 26493 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f7d7762e5d4f7f976683e00a49050f in term 1.
I20260620 07:16:16.206252 26519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:16.202498 26375 master_runner.cc:186] Error getting master registration for 127.3.245.60:44405: OK, instance_id { permanent_uuid: "4a8e40ab4b23450989480630de097df2" instance_seqno: 1781939776124660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:16:16.206750 26375 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:16:16.212258 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.212472 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.213722 26572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:16.215317 26572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 07:16:16.221336 26571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:16.221616 26571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:16.229058 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.210202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.3.245.61:43443)
0620 07:16:16.210324 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.210328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.217171 (+ 6843us) server_negotiation.cc:407] Connection header received
0620 07:16:16.217314 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.217317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.217372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.217446 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.220720 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.221415 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.222315 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.222661 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.222747 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.224921 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.224924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.224928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.224956 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.226964 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.227555 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.227562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.227567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.227656 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.227989 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.227992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.227994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.228205 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.228315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.228770 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.228897 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:16:16.229058 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.209314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.3.245.62:41105)
0620 07:16:16.209434 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.209438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.216941 (+ 7503us) server_negotiation.cc:407] Connection header received
0620 07:16:16.217743 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.217747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.217798 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.217858 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.219062 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.219780 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.222565 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.222876 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.222947 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.224921 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.224924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.224927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.224955 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.227126 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.227586 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.227590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.227591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.227667 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.227997 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.228000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.228002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.228205 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.228315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.228770 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.228891 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 07:16:16.232661 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:16.210325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.3.245.60:44405)
0620 07:16:16.210440 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.210445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.217509 (+ 7064us) server_negotiation.cc:407] Connection header received
0620 07:16:16.217613 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.217617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.217668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.217756 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.224930 (+ 7174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.225702 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.226926 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:16.227291 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:16.227368 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:16.229067 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:16.229071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:16.229074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:16.229112 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:16.230856 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.231346 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.231351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.231353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.231441 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:16.231760 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:16.231764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:16.231765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:16.231942 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:16.232095 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:16.232327 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 07:16:16.232445 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:16:16.242172 26493 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [term 1 FOLLOWER]: Refusing update from remote peer 00f7d7762e5d4f7f976683e00a49050f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:16.242324 26417 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [term 1 FOLLOWER]: Refusing update from remote peer 00f7d7762e5d4f7f976683e00a49050f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:16.243000 26526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:16.243479 26526 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:16.247439 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f7d7762e5d4f7f976683e00a49050f. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.247576 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.251669 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f7d7762e5d4f7f976683e00a49050f. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.251830 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.259065 26554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.259222 26554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:16.260138 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.260267 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.266242 26526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.266412 26526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:16.267660 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00f7d7762e5d4f7f976683e00a49050f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f7d7762e5d4f7f976683e00a49050f" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 41105 } } peers { permanent_uuid: "b3489711028c4bd7a26100876df55d05" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43443 } } peers { permanent_uuid: "4a8e40ab4b23450989480630de097df2" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 44405 } } }
I20260620 07:16:16.267796 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:16.271478 26580 mvcc.cc:205] Tried to move back new op lower bound from 7298825323440558080 to 7298825323313061888. Current Snapshot: MvccSnapshot[applied={T|T < 7298825323440558080}]
I20260620 07:16:16.271694 26578 mvcc.cc:205] Tried to move back new op lower bound from 7298825323440558080 to 7298825323313061888. Current Snapshot: MvccSnapshot[applied={T|T < 7298825323440558080}]
I20260620 07:16:16.275079 26572 catalog_manager.cc:1347] Generated new cluster ID: e1c8bda44fbb47b9a01f7c34dd34878e
I20260620 07:16:16.275164 26572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:16.305874 26572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:16.307464 26572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:16.328709 26572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f7d7762e5d4f7f976683e00a49050f: Generated new TSK 0
I20260620 07:16:16.331776 26572 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:16:16.371824 26372 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.369092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.3.245.62:41105)
0620 07:16:16.369225 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.369231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.370718 (+ 1487us) server_negotiation.cc:407] Connection header received
0620 07:16:16.370919 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.370923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.370976 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.371029 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.371682 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53134: BlockingRecv error: recv got EOF from 127.0.0.1:53134 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 07:16:16.373230 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.369825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.3.245.61:43443)
0620 07:16:16.369935 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.369941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.372302 (+ 2361us) server_negotiation.cc:407] Connection header received
0620 07:16:16.372467 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.372470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.372517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.372566 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.373108 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47302: BlockingRecv error: recv got EOF from 127.0.0.1:47302 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 07:16:16.374549 26524 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.370358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.3.245.60:44405)
0620 07:16:16.370480 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.370486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.373701 (+ 3215us) server_negotiation.cc:407] Connection header received
0620 07:16:16.373866 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.373869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.373920 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.373973 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.374440 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53784: BlockingRecv error: recv got EOF from 127.0.0.1:53784 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 07:16:16.379186 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.377548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.3.245.61:43443)
0620 07:16:16.377666 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.377670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.378145 (+ 475us) server_negotiation.cc:407] Connection header received
0620 07:16:16.378383 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.378387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.378468 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.378523 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.379076 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47318: BlockingRecv error: recv got EOF from 127.0.0.1:47318 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 07:16:16.384754 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.382928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.3.245.61:43443)
0620 07:16:16.383040 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.383045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.383677 (+ 632us) server_negotiation.cc:407] Connection header received
0620 07:16:16.383926 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.383931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.384002 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.384057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.384591 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47322: BlockingRecv error: recv got EOF from 127.0.0.1:47322 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 07:16:16.391157 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.389403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.3.245.61:43443)
0620 07:16:16.389521 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.389526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.390161 (+ 635us) server_negotiation.cc:407] Connection header received
0620 07:16:16.390371 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.390374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.390430 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.390484 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.391039 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47332: BlockingRecv error: recv got EOF from 127.0.0.1:47332 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:16:16.400511 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.398578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.3.245.61:43443)
0620 07:16:16.398763 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.398768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.399411 (+ 643us) server_negotiation.cc:407] Connection header received
0620 07:16:16.399666 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.399670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.399731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.399788 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.400386 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47336: BlockingRecv error: recv got EOF from 127.0.0.1:47336 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 07:16:16.412599 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.411233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.3.245.61:43443)
0620 07:16:16.411357 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.411362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.411620 (+ 258us) server_negotiation.cc:407] Connection header received
0620 07:16:16.411784 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.411788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.411843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.411941 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.412477 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47346: BlockingRecv error: recv got EOF from 127.0.0.1:47346 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 07:16:16.432106 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.430896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.3.245.61:43443)
0620 07:16:16.431042 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.431047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.431196 (+ 149us) server_negotiation.cc:407] Connection header received
0620 07:16:16.431311 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.431314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.431370 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.431467 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.431984 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47360: BlockingRecv error: recv got EOF from 127.0.0.1:47360 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:16:16.467288 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.466120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.3.245.61:43443)
0620 07:16:16.466273 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.466277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.466456 (+ 179us) server_negotiation.cc:407] Connection header received
0620 07:16:16.466573 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.466576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.466656 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.466730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.467157 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47366: BlockingRecv error: recv got EOF from 127.0.0.1:47366 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:16:16.534703 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.533508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.3.245.61:43443)
0620 07:16:16.533674 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.533679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.533785 (+ 106us) server_negotiation.cc:407] Connection header received
0620 07:16:16.533959 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.533962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.534019 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.534109 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.534572 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47376: BlockingRecv error: recv got EOF from 127.0.0.1:47376 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:16:16.666096 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.664830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.3.245.61:43443)
0620 07:16:16.664995 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.665000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.665139 (+ 139us) server_negotiation.cc:407] Connection header received
0620 07:16:16.665360 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.665364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.665422 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.665511 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.665972 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47382: BlockingRecv error: recv got EOF from 127.0.0.1:47382 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:16:16.925731 26447 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:16.924458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.3.245.61:43443)
0620 07:16:16.924707 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:16.924711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:16.924728 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:16:16.924841 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:16.924843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:16.924892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:16.924993 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:16.925543 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47386: BlockingRecv error: recv got EOF from 127.0.0.1:47386 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 07:16:17.192466 26551 catalog_manager.cc:1259] Loaded cluster ID: e1c8bda44fbb47b9a01f7c34dd34878e
I20260620 07:16:17.192602 26551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: loading cluster ID for follower catalog manager: success
I20260620 07:16:17.194397 26551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: acquiring CA information for follower catalog manager: success
I20260620 07:16:17.195582 26551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3489711028c4bd7a26100876df55d05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:16:17.223032 26571 catalog_manager.cc:1259] Loaded cluster ID: e1c8bda44fbb47b9a01f7c34dd34878e
I20260620 07:16:17.223184 26571 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: loading cluster ID for follower catalog manager: success
I20260620 07:16:17.224877 26571 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: acquiring CA information for follower catalog manager: success
I20260620 07:16:17.226024 26571 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a8e40ab4b23450989480630de097df2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:16:17.444432 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:17.440088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.3.245.61:43443)
0620 07:16:17.440361 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:17.440370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:17.440438 (+ 68us) server_negotiation.cc:407] Connection header received
0620 07:16:17.440639 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:17.440644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:17.440798 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:17.440929 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:17.441207 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.441811 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.442775 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.443034 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.443165 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:17.443510 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:17.444000 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:17.444144 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:17.444194 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 07:16:17.444255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260620 07:16:17.451565 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:17.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53792 (local address 127.3.245.60:44405)
0620 07:16:17.448036 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:17.448039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:17.448052 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:17.448195 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:17.448198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:17.448312 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:17.448408 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:17.448670 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.449187 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.450066 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.450302 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.450393 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:17.450705 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:17.451136 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:17.451231 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:17.451323 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 07:16:17.451370 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260620 07:16:17.459708 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:17.455290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.3.245.61:43443)
0620 07:16:17.455432 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:17.455437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:17.455492 (+ 55us) server_negotiation.cc:407] Connection header received
0620 07:16:17.455760 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:17.455764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:17.455831 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:17.455884 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:17.456268 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.457033 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.458425 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.458758 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.458858 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:17.459144 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:17.459236 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:17.459291 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:17.459495 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:16:17.459555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:16:17.459717 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:17.454926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.3.245.62:41105)
0620 07:16:17.455204 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:17.455210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:17.455246 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:16:17.455351 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:17.455358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:17.455540 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:17.455765 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:17.456144 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.456866 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.458007 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.458322 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.458476 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:17.458895 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:17.459271 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:17.459408 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:17.459495 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 07:16:17.459555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260620 07:16:17.460199 26655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:17.455670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.3.245.60:44405)
0620 07:16:17.455817 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:17.455821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:17.455960 (+ 139us) server_negotiation.cc:407] Connection header received
0620 07:16:17.456111 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:17.456115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:17.456179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:17.456269 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:17.456627 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.457351 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.458660 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:17.458981 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:17.459089 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:17.459332 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:17.459424 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:17.459493 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:17.460029 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 07:16:17.460088 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:16:17.463304 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26300
I20260620 07:16:17.470788 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26375
I20260620 07:16:17.477005 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26450
2026-06-20T07:16:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3184 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:16:17 dist-test-slave-ldfr krb5kdc[26666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:16:17 dist-test-slave-ldfr krb5kdc[26666](info): set up 2 sockets
Jun 20 07:16:17 dist-test-slave-ldfr krb5kdc[26666](info): commencing operation
krb5kdc: starting...
W20260620 07:16:18.553177 4052 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/test-user.keytab.
Jun 20 07:16:18 dist-test-slave-ldfr krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.245.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
Entry for principal kudu/127.3.245.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab.
I20260620 07:16:18.675839 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34611
--webserver_interface=127.3.245.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:18.787187 26679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:18.787557 26679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:18.787643 26679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:18.787724 26679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:16:18.791821 26679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:18.791925 26679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:18.791965 26679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:18.792001 26679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:18.792037 26679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:18.797463 26679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.62.keytab
--principal=kudu/127.3.245.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.62:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.245.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:18.798648 26679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:18.799770 26679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:18.808830 26687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:18.808851 26685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:18.808993 26679 server_base.cc:1034] running on GCE node
W20260620 07:16:18.808830 26684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:18.809738 26679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:18.811021 26679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:18.812250 26679 hybrid_clock.cc:648] HybridClock initialized: now 1781939778812215 us; error 55 us; skew 500 ppm
Jun 20 07:16:18 dist-test-slave-ldfr krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:18.815421 26679 init.cc:377] Logged in from keytab as kudu/127.3.245.62@KRBTEST.COM (short username kudu)
I20260620 07:16:18.816376 26679 webserver.cc:460] Webserver started at http://127.3.245.62:37495/ using document root <none> and password file <none>
I20260620 07:16:18.816992 26679 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:18.817059 26679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:18.817236 26679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:18.819284 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data/instance:
uuid: "5a2cc37e58e9494f83d1f18abbf310fd"
format_stamp: "Formatted at 2026-06-20 07:16:18 on dist-test-slave-ldfr"
server_key: "d6147ffba04e6d1161c9e8a6ef3e6b9f"
server_key_iv: "ccfe82dde5c61479941d94c0176dd75f"
server_key_version: "encryptionkey@0"
I20260620 07:16:18.819813 26679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance:
uuid: "5a2cc37e58e9494f83d1f18abbf310fd"
format_stamp: "Formatted at 2026-06-20 07:16:18 on dist-test-slave-ldfr"
server_key: "d6147ffba04e6d1161c9e8a6ef3e6b9f"
server_key_iv: "ccfe82dde5c61479941d94c0176dd75f"
server_key_version: "encryptionkey@0"
I20260620 07:16:18.823670 26679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:16:18.826151 26694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:18.827457 26679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:18.827590 26679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal
uuid: "5a2cc37e58e9494f83d1f18abbf310fd"
format_stamp: "Formatted at 2026-06-20 07:16:18 on dist-test-slave-ldfr"
server_key: "d6147ffba04e6d1161c9e8a6ef3e6b9f"
server_key_iv: "ccfe82dde5c61479941d94c0176dd75f"
server_key_version: "encryptionkey@0"
I20260620 07:16:18.827708 26679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:18.843026 26679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:18.846566 26679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:18.846777 26679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:18.854055 26679 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.62:34611
I20260620 07:16:18.854399 26746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.62:34611 every 8 connection(s)
I20260620 07:16:18.855080 26679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/data/info.pb
I20260620 07:16:18.857986 26747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:18.861899 26747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:18.862334 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26679
I20260620 07:16:18.862454 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-0/wal/instance
I20260620 07:16:18.862763 4052 external_mini_cluster.cc:1428] Setting key fc3e55d18a64473b4be3c28cc51441b5
WARNING: no policy specified for kudu/127.3.245.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.61@KRBTEST.COM" created.
Jun 20 07:16:18 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:18.880292 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.864804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.3.245.62:34611)
0620 07:16:18.865393 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:18.865410 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:18.865614 (+ 204us) server_negotiation.cc:407] Connection header received
0620 07:16:18.866722 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:18.866731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:18.867012 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:18.867244 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:18.868066 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:18.869628 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:18.870849 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:18.871462 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:18.871648 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:18.875228 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:18.875234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:18.875248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:18.875270 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:18.877066 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:18.877573 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:18.877585 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:18.877586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:18.877727 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:18.878142 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:18.878145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:18.878146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:18.878555 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:18.878669 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:18.879196 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 07:16:18.879450 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":464,"spinlock_wait_cycles":47360,"thread_start_us":160,"threads_started":1}
I20260620 07:16:18.880290 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.864804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34611 (local address 127.0.0.1:41214)
0620 07:16:18.865427 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:18.865487 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:18.866396 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:18.867245 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:18.867250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:18.867600 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:18.867909 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:18.867923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:18.869891 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:18.869897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:18.870718 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:18.870728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:18.871648 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:18.871652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:18.871766 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:18.872417 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:18.872434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:18.875054 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:18.877192 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:18.877201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:18.877211 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:18.877451 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:18.877858 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:18.877864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:18.877868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:18.878033 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:18.878690 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:18.878699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:18.878895 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 07:16:18.879126 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 07:16:18.879548 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":77,"thread_start_us":160,"threads_started":1}
I20260620 07:16:18.882691 26747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:18.883687 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.883349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60000)
0620 07:16:18.883536 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:18.883611 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:16:18.883924 26696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
I20260620 07:16:18.884892 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.884700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60008)
0620 07:16:18.884789 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:18.884840 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 07:16:18.885075 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43297: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
Entry for principal kudu/127.3.245.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab.
I20260620 07:16:18.891746 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43297
--webserver_interface=127.3.245.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:16:18.934334 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } attempt: 1
I20260620 07:16:18.935235 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.934886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60012)
0620 07:16:18.935062 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:18.935141 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 07:16:18.936053 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:18.935807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60020)
0620 07:16:18.935918 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:18.935979 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:16:18.936213 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43297: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
W20260620 07:16:19.003412 26754 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:19.003717 26754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:19.003741 26754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:19.003820 26754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:16:19.004417 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } attempt: 2
I20260620 07:16:19.005275 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.004978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60022)
0620 07:16:19.005138 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.005202 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:16:19.006001 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.005806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60036)
0620 07:16:19.005903 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.005948 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 07:16:19.006172 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.61:43297: Network error: Client connection negotiation failed: client connection to 127.3.245.61:43297: connect: Connection refused (error 111)
W20260620 07:16:19.008570 26754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:19.008675 26754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:19.008742 26754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:19.008781 26754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:19.008814 26754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:19.014364 26754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.61.keytab
--principal=kudu/127.3.245.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.61:43297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.245.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:19.015443 26754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:19.016484 26754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:16:19.024914 26759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:19.024952 26760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:19.025115 26754 server_base.cc:1034] running on GCE node
W20260620 07:16:19.024981 26762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:19.025774 26754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:19.026979 26754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:19.028160 26754 hybrid_clock.cc:648] HybridClock initialized: now 1781939779028143 us; error 33 us; skew 500 ppm
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:19.030966 26754 init.cc:377] Logged in from keytab as kudu/127.3.245.61@KRBTEST.COM (short username kudu)
I20260620 07:16:19.031961 26754 webserver.cc:460] Webserver started at http://127.3.245.61:42717/ using document root <none> and password file <none>
I20260620 07:16:19.032577 26754 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:19.032711 26754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:19.032948 26754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:19.034996 26754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data/instance:
uuid: "b8ec69de2376421882c2ed54c62a32ec"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "596f0233e15841a10fb0b13c1e0d9408"
server_key_iv: "2d0101a0e3a4fca8991b2e1a96d9286b"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.035566 26754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal/instance:
uuid: "b8ec69de2376421882c2ed54c62a32ec"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "596f0233e15841a10fb0b13c1e0d9408"
server_key_iv: "2d0101a0e3a4fca8991b2e1a96d9286b"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.039289 26754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260620 07:16:19.041937 26769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:19.043215 26754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 07:16:19.043366 26754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal
uuid: "b8ec69de2376421882c2ed54c62a32ec"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "596f0233e15841a10fb0b13c1e0d9408"
server_key_iv: "2d0101a0e3a4fca8991b2e1a96d9286b"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.043489 26754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:19.069758 26754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:19.072901 26754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:19.073120 26754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:19.080107 26754 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.61:43297
I20260620 07:16:19.080396 26821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.61:43297 every 8 connection(s)
I20260620 07:16:19.081120 26754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/data/info.pb
I20260620 07:16:19.083897 26822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:19.087666 26822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.088512 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26754
I20260620 07:16:19.088675 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-1/wal/instance
I20260620 07:16:19.088977 4052 external_mini_cluster.cc:1428] Setting key 73452819cb726b8b259a9b163427be22
WARNING: no policy specified for kudu/127.3.245.60@KRBTEST.COM; defaulting to no policy
I20260620 07:16:19.097337 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.245.60@KRBTEST.COM" created.
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:19.106068 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.089911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.3.245.62:34611)
0620 07:16:19.090074 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.090079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.090965 (+ 886us) server_negotiation.cc:407] Connection header received
0620 07:16:19.091914 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.091921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.091988 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.092097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.093519 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.094482 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.095824 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.096220 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.096353 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.100898 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.100905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.100911 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.100947 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.103315 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.104005 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.104008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.104010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.104116 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.104573 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.104577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.104580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.104840 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.104957 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.105755 (+ 798us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.105881 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:16:19.107018 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.090299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34611 (local address 127.0.0.1:41220)
0620 07:16:19.090788 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.090841 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.091706 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.092315 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.092323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.092819 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.093342 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.093357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.094650 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.094653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.095682 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.095692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.096396 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.096399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.096559 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.097368 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.097391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.100744 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.103447 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.103457 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.103480 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.103887 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.104213 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.104219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.104225 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.104436 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.104958 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.104966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.105481 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.105697 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.106074 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260620 07:16:19.108294 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.097837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60038)
0620 07:16:19.097989 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.098006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.098117 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.098772 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.098775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.098926 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.099030 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.099036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.100179 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.100185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.100969 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.100979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.101784 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.101790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.101930 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.102407 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.102426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.103881 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.106467 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.106472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.106476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.106787 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.107181 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.107187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.107191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.107285 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.107883 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.107885 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.107991 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.108078 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.108186 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:16:19.108570 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.098035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.3.245.61:43297)
0620 07:16:19.098388 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.098397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.098418 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:16:19.098453 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.098457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.098617 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.098753 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.099138 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.100015 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.101083 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.101644 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.101768 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.104243 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.104252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.104264 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.104286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.106321 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.106949 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.106956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.106959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.107056 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.107421 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.107423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.107426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.107760 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.107909 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.108201 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.108372 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":142,"threads_started":1}
I20260620 07:16:19.109213 26822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.110605 26747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.111337 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.111094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42072)
0620 07:16:19.111218 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.111276 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:16:19.112073 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.111800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42084)
0620 07:16:19.111976 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.112023 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 07:16:19.112335 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:19.118705 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.109754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60046)
0620 07:16:19.109943 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.109957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.110030 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.110670 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.110675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.110846 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.110975 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.110986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.111924 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.111930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.112692 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.112703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.113159 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.113163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.113296 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.113735 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.113747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.115038 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.116906 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.116909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.116910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.117201 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.117448 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.117450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.117451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.117569 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.117983 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.117986 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.118115 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.118246 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.118403 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":40}
I20260620 07:16:19.118716 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.109770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.3.245.61:43297)
0620 07:16:19.110440 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.110443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.110454 (+ 11us) server_negotiation.cc:407] Connection header received
0620 07:16:19.110480 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.110482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.110550 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.110635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.111086 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.111807 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.112797 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.113039 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.113160 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.115162 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.115165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.115167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.115187 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.116798 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.117289 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.117292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.117293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.117361 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.117670 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.117674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.117676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.117880 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.117983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.118305 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.118456 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":618,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
Entry for principal kudu/127.3.245.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab.
I20260620 07:16:19.119489 26822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.120294 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.119979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42086)
0620 07:16:19.120121 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.120219 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:16:19.120301 4052 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
/tmp/dist-test-taskVl0LDo/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:41517
--webserver_interface=127.3.245.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.245.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:16:19.121057 26771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
I20260620 07:16:19.121971 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.121767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42094)
0620 07:16:19.121873 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.121921 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:16:19.122190 26822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
I20260620 07:16:19.143528 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 1
I20260620 07:16:19.144384 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.144094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42102)
0620 07:16:19.144238 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.144302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:16:19.145208 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.144965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42104)
0620 07:16:19.145077 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.145154 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:16:19.145383 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
I20260620 07:16:19.171478 26822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 1
I20260620 07:16:19.172636 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.172181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42118)
0620 07:16:19.172412 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.172496 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:16:19.173723 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.173427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42120)
0620 07:16:19.173562 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.173626 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:16:19.173972 26822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
I20260620 07:16:19.220572 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 2
I20260620 07:16:19.221475 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.221182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42128)
0620 07:16:19.221332 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.221399 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 07:16:19.222280 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.222019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42140)
0620 07:16:19.222175 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.222225 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:16:19.222398 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
W20260620 07:16:19.239269 26830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:16:19.239638 26830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:16:19.239701 26830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:16:19.239778 26830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 07:16:19.242147 26822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 2
I20260620 07:16:19.242988 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.242704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42154)
0620 07:16:19.242851 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.242913 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:16:19.243728 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.243515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42158)
0620 07:16:19.243630 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.243679 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 07:16:19.243851 26822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
W20260620 07:16:19.244369 26830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:16:19.244445 26830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:16:19.244489 26830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:16:19.244537 26830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:16:19.244571 26830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:16:19.250138 26830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.245.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.245.62:34611,127.3.245.61:43297,127.3.245.60:41517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/krb5kdc/kudu_127.3.245.60.keytab
--principal=kudu/127.3.245.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.245.60:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.245.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 06:29:27 UTC on ccycloud.quasar-mcsiyf.root.comops.site
build id 4457016
I20260620 07:16:19.251283 26830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:16:19.252321 26830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:16:19.261328 26830 server_base.cc:1034] running on GCE node
W20260620 07:16:19.261343 26835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:19.261343 26838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:16:19.261497 26836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:16:19.262233 26830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:16:19.263370 26830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:16:19.264586 26830 hybrid_clock.cc:648] HybridClock initialized: now 1781939779264567 us; error 33 us; skew 500 ppm
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:16:19.267436 26830 init.cc:377] Logged in from keytab as kudu/127.3.245.60@KRBTEST.COM (short username kudu)
I20260620 07:16:19.268420 26830 webserver.cc:460] Webserver started at http://127.3.245.60:36287/ using document root <none> and password file <none>
I20260620 07:16:19.269053 26830 fs_manager.cc:359] Metadata directory not provided
I20260620 07:16:19.269148 26830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:16:19.269383 26830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:16:19.271598 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data/instance:
uuid: "6c7ffd2211a647439cda82b1e193dda3"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "f34961e4b249845ff617a6228570985d"
server_key_iv: "ad04e5089c4b316ede54ef57a336f72d"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.272157 26830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal/instance:
uuid: "6c7ffd2211a647439cda82b1e193dda3"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "f34961e4b249845ff617a6228570985d"
server_key_iv: "ad04e5089c4b316ede54ef57a336f72d"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.275972 26830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:16:19.278538 26845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:16:19.279917 26830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:16:19.280066 26830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data,/tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal
uuid: "6c7ffd2211a647439cda82b1e193dda3"
format_stamp: "Formatted at 2026-06-20 07:16:19 on dist-test-slave-ldfr"
server_key: "f34961e4b249845ff617a6228570985d"
server_key_iv: "ad04e5089c4b316ede54ef57a336f72d"
server_key_version: "encryptionkey@0"
I20260620 07:16:19.280195 26830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:16:19.317559 26830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:16:19.321130 26830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:16:19.321352 26830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:16:19.321552 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 3
I20260620 07:16:19.322352 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.322017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42164)
0620 07:16:19.322205 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.322275 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:16:19.323206 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.322981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42178)
0620 07:16:19.323089 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.323133 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 07:16:19.323429 26747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.245.60:41517: Network error: Client connection negotiation failed: client connection to 127.3.245.60:41517: connect: Connection refused (error 111)
I20260620 07:16:19.328704 26830 rpc_server.cc:307] RPC server started. Bound to: 127.3.245.60:41517
I20260620 07:16:19.328989 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.245.60:41517 every 8 connection(s)
I20260620 07:16:19.329730 26830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/data/info.pb
I20260620 07:16:19.332975 26898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:16:19.335013 26822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 3
I20260620 07:16:19.337765 4052 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu as pid 26830
I20260620 07:16:19.337177 26898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.337965 4052 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVl0LDo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781939488540940-4052-0/minicluster-data/master-2/wal/instance
I20260620 07:16:19.338227 4052 external_mini_cluster.cc:1428] Setting key d9634bce9863ae75dc3d8c08af5ab277
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.61@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:19.350262 26825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.335509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42182)
0620 07:16:19.335660 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.335678 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.335796 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.338495 (+ 2699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.338499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.338722 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.338857 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.338865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.340924 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.340928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.341648 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.341660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.342683 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.342687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.342771 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.343362 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.343378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.344976 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.347859 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.347866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.347868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.348172 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.348734 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.348739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.348742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.348848 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.349681 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.349686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.349792 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.349896 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.350038 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 07:16:19.350485 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.339606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.3.245.62:34611)
0620 07:16:19.339718 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.339723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.340126 (+ 403us) server_negotiation.cc:407] Connection header received
0620 07:16:19.340306 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.340309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.340352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.340439 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.341292 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.341932 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.343116 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.343448 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.343571 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.346857 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.346863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.346867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.346900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.348483 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.349047 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.349051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.349053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.349133 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.349416 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.349419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.349421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.349597 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.349723 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.350201 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.350322 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:16:19.351830 26902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.339042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.3.245.60:41517)
0620 07:16:19.339400 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.339409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.339424 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:16:19.339659 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.339663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.339708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.339787 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.340761 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.341456 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.342569 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.343046 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.343139 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.346366 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.346369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.346371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.346396 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.349002 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.349489 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.349493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.349495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.349571 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.349874 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.349879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.349881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.350050 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.350229 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.350809 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.351067 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":181,"server-negotiator.queue_time_us":293,"thread_start_us":146,"threads_started":1}
I20260620 07:16:19.351832 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.339399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.62:34611 (local address 127.0.0.1:41234)
0620 07:16:19.339978 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.340029 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.340191 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.340489 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.340495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.341018 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.341145 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.341163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.342053 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.342057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.342943 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.342953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.343827 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.343835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.343923 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.344482 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.344503 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.346709 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.348599 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.348607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.348637 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.348945 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.349233 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.349236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.349238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.349323 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.349778 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.349785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.349940 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.350674 (+ 734us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.351148 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":267,"thread_start_us":85,"threads_started":1}
I20260620 07:16:19.351832 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.336258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.3.245.60:41517)
0620 07:16:19.336731 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.336752 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.336795 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:16:19.337631 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.337656 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.338135 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.338525 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.339063 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.340792 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.341766 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.342505 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.342734 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.345143 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.345151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.345167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.345200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.347473 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.348419 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.348427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.348434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.348565 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.348986 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.348989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.348992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.349545 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.349677 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.350650 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.350970 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260620 07:16:19.353223 26898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.356370 26822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: Bootstrap starting.
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:19.358919 26822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:19.359858 26822 log.cc:824] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.62@KRBTEST.COM
I20260620 07:16:19.362356 26822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: No bootstrap required, opened a new log
I20260620 07:16:19.362717 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.353714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.61:43297 (local address 127.0.0.1:60048)
0620 07:16:19.354024 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.354039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.354125 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.354367 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.354373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.354553 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.354677 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.354684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.355604 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.355607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.356405 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.356418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.357032 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.357035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.357100 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.357609 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.357623 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.359035 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.361050 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.361053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.361056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.361299 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.361622 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.361625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.361627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.361716 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.362129 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.362132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.362234 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.362340 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.362509 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260620 07:16:19.363380 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.353825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.3.245.61:43297)
0620 07:16:19.353931 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.353936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.354087 (+ 151us) server_negotiation.cc:407] Connection header received
0620 07:16:19.354201 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.354203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.354252 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.354331 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.354775 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.355491 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.356537 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.356844 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.356904 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.359151 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.359154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.359157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.359181 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.360921 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.361403 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.361406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.361409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.361509 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.361813 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.361817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.361819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.362029 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.362138 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.362731 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.363280 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 07:16:19.364221 26898 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:16:19.365895 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.354567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.3.245.62:34611)
0620 07:16:19.354678 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.354682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.355605 (+ 923us) server_negotiation.cc:407] Connection header received
0620 07:16:19.355644 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.355646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.355699 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.355774 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.356486 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.357204 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.358568 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.358968 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.359109 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.362107 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.362111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.362113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.362138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.363751 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.364240 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.364247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.364249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.364350 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.364847 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.364850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.364852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.365125 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.365354 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.365581 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.365720 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:16:19.367034 26822 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.367275 26822 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ec69de2376421882c2ed54c62a32ec, State: Initialized, Role: FOLLOWER
I20260620 07:16:19.367627 26822 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.368153 26905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.368314 26905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.369112 26822 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.60@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.245.61@KRBTEST.COM
I20260620 07:16:19.374886 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.364810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:42190)
0620 07:16:19.365433 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.365449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.365562 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.365895 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.365899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.366099 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.366198 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.366207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.367204 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.367207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.367975 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.367986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.368866 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.368869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.368939 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.369445 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.369462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.371148 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.373272 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.373275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.373277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.373514 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.373865 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.373867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.373869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.373951 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.374405 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.374408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.374485 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.374573 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.374714 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563}
W20260620 07:16:19.375499 26916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:19.375584 26916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:19.375736 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.365047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.3.245.60:41517)
0620 07:16:19.365180 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.365187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.365645 (+ 458us) server_negotiation.cc:407] Connection header received
0620 07:16:19.365679 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.365684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.365727 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.365797 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.366333 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.367082 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.368234 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.368545 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.368678 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.371295 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.371299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.371302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.371327 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.373163 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.373612 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.373618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.373623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.373715 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.374036 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.374038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.374040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.374282 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.374374 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.375428 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.375573 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:16:19.373601 26754 master_runner.cc:186] Error getting master registration for 127.3.245.62:34611: OK, instance_id { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" instance_seqno: 1781939778851858 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:16:19.377708 26754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:16:19.378667 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.367255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.3.245.61:43297)
0620 07:16:19.367375 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.367379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.367392 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:16:19.367443 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.367446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.367482 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.367575 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.368333 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.368947 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.369894 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.370256 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.373716 (+ 3460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.374360 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.374364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.374367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.374396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.376175 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.376740 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.376744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.376746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.376841 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.377210 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.377214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.377215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.377441 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.377619 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.378396 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.378518 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:19.379201 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: Bootstrap starting.
I20260620 07:16:19.382117 26898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:19.382131 4052 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:16:19.383064 26898 log.cc:824] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:19.383363 26721 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5a2cc37e58e9494f83d1f18abbf310fd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41244
I20260620 07:16:19.385633 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: No bootstrap required, opened a new log
I20260620 07:16:19.389871 26898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.390143 26898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7ffd2211a647439cda82b1e193dda3, State: Initialized, Role: FOLLOWER
I20260620 07:16:19.390506 26898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.391121 26919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.391345 26919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.391947 26898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:16:19.397637 26930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:19.397719 26930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:19.395898 26830 master_runner.cc:186] Error getting master registration for 127.3.245.62:34611: OK, instance_id { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" instance_seqno: 1781939778851858 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:16:19.399150 26830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:16:19.487620 26747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } attempt: 4
Jun 20 07:16:19 dist-test-slave-ldfr krb5kdc[26666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781939778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.245.62@KRBTEST.COM for kudu/127.3.245.60@KRBTEST.COM
I20260620 07:16:19.498549 26750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.488228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.245.60:41517 (local address 127.0.0.1:49804)
0620 07:16:19.488413 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 07:16:19.488434 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:16:19.488549 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:16:19.489133 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:16:19.489136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:16:19.489367 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:16:19.489486 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.489492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.490509 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.490513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.491341 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:16:19.491352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.492117 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.492120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:16:19.492232 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.492918 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:16:19.492934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:16:19.494602 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:16:19.496751 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.496754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.496757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.497011 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.497365 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:16:19.497369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:16:19.497370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:16:19.497475 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:16:19.497996 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:16:19.498000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:16:19.498101 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 07:16:19.498222 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 07:16:19.498368 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 07:16:19.499114 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.488744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.3.245.60:41517)
0620 07:16:19.488867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.488872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.488887 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:16:19.488935 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.488938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.488992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.489074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.489599 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.490377 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.491487 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.491814 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.491947 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.494773 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.494777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.494780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.494808 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.496602 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.497133 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.497137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.497139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.497251 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.497588 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.497591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.497593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.497824 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.497923 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.498842 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.498965 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:16:19.502542 26747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: Bootstrap starting.
I20260620 07:16:19.505622 26747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: Neither blocks nor log segments found. Creating new log.
I20260620 07:16:19.506587 26747 log.cc:824] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: Log is configured to *not* fsync() on all Append() calls
I20260620 07:16:19.509001 26747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: No bootstrap required, opened a new log
I20260620 07:16:19.512207 26747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.512446 26747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2cc37e58e9494f83d1f18abbf310fd, State: Initialized, Role: FOLLOWER
I20260620 07:16:19.512846 26747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.513547 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.513818 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.514348 26747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:16:19.514405 26721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:16:19.514503 26721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:19.515699 26721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.517652 26796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8ec69de2376421882c2ed54c62a32ec"
I20260620 07:16:19.517920 26796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:19.516656 26721 leader_election.cc:288] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [CANDIDATE]: Term 1 election: Requested vote from peers b8ec69de2376421882c2ed54c62a32ec (127.3.245.61:43297), 6c7ffd2211a647439cda82b1e193dda3 (127.3.245.60:41517)
I20260620 07:16:19.519347 26796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cc37e58e9494f83d1f18abbf310fd in term 1.
I20260620 07:16:19.519806 26872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c7ffd2211a647439cda82b1e193dda3"
I20260620 07:16:19.520025 26872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:16:19.521276 26872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a2cc37e58e9494f83d1f18abbf310fd in term 1.
I20260620 07:16:19.521732 26696 leader_election.cc:302] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a2cc37e58e9494f83d1f18abbf310fd, 6c7ffd2211a647439cda82b1e193dda3; no voters:
I20260620 07:16:19.522087 26933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:16:19.522341 26933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [term 1 LEADER]: Becoming Leader. State: Replica: 5a2cc37e58e9494f83d1f18abbf310fd, State: Running, Role: LEADER
I20260620 07:16:19.522718 26933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } }
I20260620 07:16:19.526314 26679 master_runner.cc:263] No leader master found from master 5a2cc37e58e9494f83d1f18abbf310fd
I20260620 07:16:19.526383 26679 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:16:19.528023 26944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:16:19.528152 26944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:16:19.528304 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2cc37e58e9494f83d1f18abbf310fd. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.528451 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:19.540098 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.529036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.3.245.61:43297)
0620 07:16:19.529163 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.529168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.530755 (+ 1587us) server_negotiation.cc:407] Connection header received
0620 07:16:19.530949 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.530953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.531014 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.531064 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.532179 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.532946 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.534142 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.534491 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.534611 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.536419 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.536423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.536426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.536455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.538198 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.538720 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.538724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.538726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.538811 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.539124 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.539128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.539130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.539347 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.539449 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.539813 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.539931 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:19.542757 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.530180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.3.245.60:41517)
0620 07:16:19.530306 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.530310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.533951 (+ 3641us) server_negotiation.cc:407] Connection header received
0620 07:16:19.534135 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.534139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.534195 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.534236 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.535066 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.535807 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.536927 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.537251 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.537331 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.538934 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.538938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.538941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.538980 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.540724 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.541447 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.541451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.541453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.541537 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.541843 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.541847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.541848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.542078 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.542245 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.542465 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.542584 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:16:19.544070 26952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:16:19.545136 26952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:16:19.548789 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:19.529502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.3.245.62:34611)
0620 07:16:19.529610 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.529615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.531932 (+ 2317us) server_negotiation.cc:407] Connection header received
0620 07:16:19.532713 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.532717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.532774 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.532821 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.540746 (+ 7925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.541538 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.542502 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:19.542874 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:19.542961 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:19.545103 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:16:19.545107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:16:19.545110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:16:19.545138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:16:19.546899 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.547378 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.547382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.547384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.547466 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:16:19.547790 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:16:19.547793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:16:19.547795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:16:19.548018 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:16:19.548132 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:19.548493 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 07:16:19.548642 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:16:19.558912 26872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [term 1 FOLLOWER]: Refusing update from remote peer 5a2cc37e58e9494f83d1f18abbf310fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:19.559401 26796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [term 1 FOLLOWER]: Refusing update from remote peer 5a2cc37e58e9494f83d1f18abbf310fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:16:19.559646 26933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:19.560216 26933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:16:19.563767 26919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2cc37e58e9494f83d1f18abbf310fd. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.563810 26905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a2cc37e58e9494f83d1f18abbf310fd. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.563895 26919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.563912 26905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.566969 26949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.567140 26949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:19.568287 26919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.568403 26919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.569507 26958 mvcc.cc:205] Tried to move back new op lower bound from 7298825337064476672 to 7298825336928956416. Current Snapshot: MvccSnapshot[applied={T|T < 7298825337064476672}]
I20260620 07:16:19.568867 26956 mvcc.cc:205] Tried to move back new op lower bound from 7298825337064476672 to 7298825336928956416. Current Snapshot: MvccSnapshot[applied={T|T < 7298825337064476672}]
I20260620 07:16:19.570413 26905 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.570545 26905 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:16:19.570824 26957 mvcc.cc:205] Tried to move back new op lower bound from 7298825337064476672 to 7298825336928956416. Current Snapshot: MvccSnapshot[applied={T|T < 7298825337064476672}]
I20260620 07:16:19.571398 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a2cc37e58e9494f83d1f18abbf310fd" member_type: VOTER last_known_addr { host: "127.3.245.62" port: 34611 } } peers { permanent_uuid: "b8ec69de2376421882c2ed54c62a32ec" member_type: VOTER last_known_addr { host: "127.3.245.61" port: 43297 } } peers { permanent_uuid: "6c7ffd2211a647439cda82b1e193dda3" member_type: VOTER last_known_addr { host: "127.3.245.60" port: 41517 } } }
I20260620 07:16:19.571532 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd [sys.catalog]: This master's current role is: LEADER
I20260620 07:16:19.574896 26952 catalog_manager.cc:1347] Generated new cluster ID: 4a2ff13fcb0a4bd4b7bcc8f088b49851
I20260620 07:16:19.574959 26952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:16:19.589608 26952 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:16:19.590966 26952 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:16:19.600936 26952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a2cc37e58e9494f83d1f18abbf310fd: Generated new TSK 0
I20260620 07:16:19.602216 26952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:16:19.628964 26751 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.626722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.3.245.62:34611)
0620 07:16:19.626857 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.626861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.627813 (+ 952us) server_negotiation.cc:407] Connection header received
0620 07:16:19.628030 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.628035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.628094 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.628182 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.628782 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58358: BlockingRecv error: recv got EOF from 127.0.0.1:58358 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 07:16:19.629998 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.627212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.3.245.61:43297)
0620 07:16:19.627345 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.627351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.628897 (+ 1546us) server_negotiation.cc:407] Connection header received
0620 07:16:19.629250 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.629254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.629306 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.629361 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.629877 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53638: BlockingRecv error: recv got EOF from 127.0.0.1:53638 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 07:16:19.630290 26900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.627916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.3.245.60:41517)
0620 07:16:19.628040 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.628045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.629375 (+ 1330us) server_negotiation.cc:407] Connection header received
0620 07:16:19.629561 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.629564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.629615 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.629667 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.630179 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49822: BlockingRecv error: recv got EOF from 127.0.0.1:49822 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 07:16:19.634732 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.632504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.3.245.61:43297)
0620 07:16:19.632600 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.632604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.633806 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 07:16:19.634019 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.634023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.634096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.634145 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.634629 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53642: BlockingRecv error: recv got EOF from 127.0.0.1:53642 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 07:16:19.639631 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.638191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.3.245.61:43297)
0620 07:16:19.638315 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.638322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.638810 (+ 488us) server_negotiation.cc:407] Connection header received
0620 07:16:19.638912 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.638915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.638977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.639062 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.639532 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53658: BlockingRecv error: recv got EOF from 127.0.0.1:53658 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 07:16:19.644734 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.643586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.3.245.61:43297)
0620 07:16:19.643709 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.643712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.643917 (+ 205us) server_negotiation.cc:407] Connection header received
0620 07:16:19.644071 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.644075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.644124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.644223 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.644605 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53666: BlockingRecv error: recv got EOF from 127.0.0.1:53666 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 07:16:19.651417 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.650429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.3.245.61:43297)
0620 07:16:19.650693 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.650698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.650714 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:16:19.650788 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.650791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.650840 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.650926 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.651317 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53670: BlockingRecv error: recv got EOF from 127.0.0.1:53670 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
W20260620 07:16:19.662819 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.661780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.3.245.61:43297)
0620 07:16:19.661915 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.661921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.662024 (+ 103us) server_negotiation.cc:407] Connection header received
0620 07:16:19.662116 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.662120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.662191 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.662306 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.662694 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53672: BlockingRecv error: recv got EOF from 127.0.0.1:53672 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 07:16:19.682166 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.680931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.3.245.61:43297)
0620 07:16:19.681138 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.681144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.681171 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:16:19.681338 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.681342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.681412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.681538 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.682024 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53674: BlockingRecv error: recv got EOF from 127.0.0.1:53674 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 07:16:19.717793 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.716597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.3.245.61:43297)
0620 07:16:19.716773 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.716779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.716867 (+ 88us) server_negotiation.cc:407] Connection header received
0620 07:16:19.716994 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.716999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.717065 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.717191 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.717644 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53684: BlockingRecv error: recv got EOF from 127.0.0.1:53684 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 07:16:19.785243 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.784006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.3.245.61:43297)
0620 07:16:19.784196 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.784202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.784354 (+ 152us) server_negotiation.cc:407] Connection header received
0620 07:16:19.784446 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.784449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.784506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.784590 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.785130 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53700: BlockingRecv error: recv got EOF from 127.0.0.1:53700 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 07:16:19.916908 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:19.915492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.3.245.61:43297)
0620 07:16:19.915666 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:19.915671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:19.915873 (+ 202us) server_negotiation.cc:407] Connection header received
0620 07:16:19.916066 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:19.916068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:19.916125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:19.916215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:19.916794 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53714: BlockingRecv error: recv got EOF from 127.0.0.1:53714 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 07:16:20.176259 26826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:16:20.175227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.3.245.61:43297)
0620 07:16:20.175387 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.175390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.175463 (+ 73us) server_negotiation.cc:407] Connection header received
0620 07:16:20.175579 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.175582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.175668 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.175753 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:16:20.176160 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53720: BlockingRecv error: recv got EOF from 127.0.0.1:53720 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:16:20.376930 26916 catalog_manager.cc:1259] Loaded cluster ID: 4a2ff13fcb0a4bd4b7bcc8f088b49851
I20260620 07:16:20.377013 26916 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: loading cluster ID for follower catalog manager: success
I20260620 07:16:20.378769 26916 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: acquiring CA information for follower catalog manager: success
I20260620 07:16:20.379981 26916 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8ec69de2376421882c2ed54c62a32ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:16:20.398991 26930 catalog_manager.cc:1259] Loaded cluster ID: 4a2ff13fcb0a4bd4b7bcc8f088b49851
I20260620 07:16:20.399076 26930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: loading cluster ID for follower catalog manager: success
I20260620 07:16:20.400781 26930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: acquiring CA information for follower catalog manager: success
I20260620 07:16:20.401969 26930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c7ffd2211a647439cda82b1e193dda3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:16:20.694438 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:20.690367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.3.245.61:43297)
0620 07:16:20.690619 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.690622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.690639 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:16:20.690771 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.690773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.690896 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.691011 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:20.691246 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.691842 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.692803 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.693086 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.693171 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:20.693459 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:20.693928 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:20.694066 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:20.694209 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 07:16:20.694261 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":109,"threads_started":1}
I20260620 07:16:20.701258 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:20.697424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.3.245.60:41517)
0620 07:16:20.697711 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.697717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.697731 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:16:20.697839 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.697842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.697952 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.698078 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:20.698326 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.698873 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.699766 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.699998 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.700105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:20.700398 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:20.700870 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:20.700999 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:20.701040 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 07:16:20.701103 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":62,"threads_started":1}
I20260620 07:16:20.708650 27034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:20.704862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.3.245.60:41517)
0620 07:16:20.704972 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.704975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.705195 (+ 220us) server_negotiation.cc:407] Connection header received
0620 07:16:20.705376 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.705379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.705434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.705488 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:20.705847 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.706582 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.707384 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.707665 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.707758 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:20.708081 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:20.708161 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:20.708227 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:20.708376 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 07:16:20.708446 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:16:20.708781 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:20.704411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.3.245.61:43297)
0620 07:16:20.704605 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.704631 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.705028 (+ 397us) server_negotiation.cc:407] Connection header received
0620 07:16:20.705192 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.705195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.705247 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.705331 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:20.705659 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.706430 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.707625 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.707922 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.708018 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:20.708272 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:20.708365 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:20.708433 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:20.708575 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 07:16:20.708653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 07:16:20.709925 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:16:20.704367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.3.245.62:34611)
0620 07:16:20.704604 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 07:16:20.704630 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 07:16:20.704709 (+ 79us) server_negotiation.cc:407] Connection header received
0620 07:16:20.704901 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:16:20.704904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:16:20.705077 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:16:20.705221 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:16:20.705708 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.706302 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.708090 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:16:20.708457 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:16:20.708568 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:16:20.709076 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:16:20.709423 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:16:20.709509 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 07:16:20.709700 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:16:20.709773 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260620 07:16:20.719746 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26679
I20260620 07:16:20.726524 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26754
I20260620 07:16:20.732834 4052 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVl0LDo/build/debug/bin/kudu with pid 26830
2026-06-20T07:16:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3255 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6440 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292186 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST